search

langchain-ai / langsmith-cookbook

https://langsmith-cookbook.vercel.app
783 stars 149 forks source link

The RAGAS example doesn't seem to work #253

Open dividor opened 3 months ago

dividor commented 3 months ago

Looking at the RAGAs sample notebook ...

https://github.com/langchain-ai/langsmith-cookbook/blob/main/testing-examples/ragas/ragas.ipynb

Running the last panel ...

results = await client.arun_on_dataset(
    dataset_name=dataset_name,
    llm_or_chain_factory=rag_bot.get_answer,
    evaluation=eval_config,
)

Gives a lot of errors ...

''' RuntimeError: There is no current event loop in thread 'asyncio_0'. Error in EvaluatorCallbackHandler.on_chain_end callback: RuntimeError("There is no current event loop in thread 'asyncio_0'.") [-> ] 1/21 '''

Here are the versions I'm using ...

''' Python 3.11.4 ragas 0.1.13 langchain 0.2.12 langchain-community 0.2.11 langchain-core 0.2.28 langchain-openai 0.1.20 langchain-text-splitters 0.2.2 langsmith 0.1.98 '''

My guess is some version issue, what are the versions used in the demo notebook please?

dividor commented 3 months ago

Actually, looking at the demo on the RAGAS site, I believe the notebook needs ...

import nest_asyncio

nest_asyncio.apply()

Then it works.