Open cdennison opened 2 weeks ago
Hello @cdennison! I am here to assist you.
Import Issues:
ModuleNotFoundError
for ragas.langchain
.Integration with LangChain:
VectorstoreIndexCreator
.Here's a revised version of your code, ensuring correct imports and usage of VectorstoreIndexCreator
:
import os
from getpass import getpass
from dotenv import load_dotenv
import nest_asyncio
from langchain_community.document_loaders import TextLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import faithfulness, answer_relevancy, answer_correctness
from datasets import Dataset
from ragas import evaluate
# Load environment variables
load_dotenv()
os.environ["OPENAI_API_KEY"] = getpass()
# Initialize OpenAI API client
api_key = os.environ.get("OPENAI_API_KEY")
openai.api_key = api_key
nest_asyncio.apply()
# Initialize LLM
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)
# Build query engine
def build_query_engine(llm):
loader = TextLoader("./notebooks_nyc_wikipedia_nyc_text.txt")
data = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
all_splits = text_splitter.split_documents(data)
vectorstore = Chroma.from_documents(documents=all_splits, embedding=OpenAIEmbeddings())
return RetrievalQA.from_chain_type(
llm,
retriever=vectorstore.as_retriever(),
return_source_documents=True
)
# Evaluation questions and answers
eval_questions = [
"What is the population of New York City as of 2020?",
"Which borough of New York City has the highest population?",
"What is the economic significance of New York City?",
"How did New York City get its name?",
"What is the significance of the Statue of Liberty in New York City?",
]
eval_answers = [
"8,804,190",
"Brooklyn",
"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
"New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
"The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]
examples = [
{"query": q, "ground_truth": [eval_answers[i]]}
for i, q in enumerate(eval_questions)
]
# Run the queries as a batch for efficiency
qa_chain = build_query_engine(llm)
predictions = qa_chain.batch(examples)
# Generate responses
def generate_responses(predictions):
answers = []
contexts = []
test_answers = []
test_questions = []
for r in predictions:
test_questions.append(r['query'])
answers.append(r['result'])
test_answers.append(str(r['ground_truth'][0]))
contexts.append([c.page_content for c in r['source_documents']])
dataset_dict = {
"question": test_questions,
"answer": answers,
"contexts": contexts,
"ground_truth": test_answers
}
return Dataset.from_dict(dataset_dict)
# Evaluate using Ragas
metrics = [faithfulness, answer_relevancy, answer_correctness]
res = evaluate(generate_responses(predictions), metrics=metrics)
print(res)
This code should address the issues with imports and ensure the correct usage of VectorstoreIndexCreator
and other components. Make sure to follow the latest documentation and updates from Ragas and LangChain for any changes or additional guidance [2][1].
To continue talking to Dosu, mention @dosu.
The existing documentation has several issues: https://docs.ragas.io/en/latest/howtos/integrations/langchain.html
Your Question Does Ragas plan to offer long terms support for LangChain (if not you can ignore the rest I'll have to abandon Ragas)? Can you please review the following code to see if it's correct? It's based on https://docs.ragas.io/en/latest/howtos/applications/compare_llms.html and https://docs.smith.langchain.com/old/cookbook/hub-examples/retrieval-qa-chain
Code Examples
ragas==0.1.14 langchain==0.2.14