search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
6.32k stars 606 forks source link

ExceptionInRunner and RuntimeWarning Issues with Ragas Library #1088

Closed Elena4750 closed 1 month ago

Elena4750 commented 1 month ago

I am encountering an issue while using the Ragas library. Has anyone encountered a similar problem? Environment: Python 3.10, latest version of Ragas library

image image (1)

dosubot[bot] commented 1 month ago

Hey @Elena4750! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's work together to tackle this issue!

Yes, the RuntimeError indicating that an event loop is already running can occur when using the evaluate function from the ragas package in Python 3.10. This is likely due to the is_async parameter being set to True by default, which attempts to run the evaluation asynchronously.

To resolve this issue, you can set the is_async parameter to False when calling the evaluate function. Here is an example:

from ragas import evaluate

# Assuming `dataset` and `metrics` are already defined
result = evaluate(dataset, metrics=metrics, is_async=False)
df = result.to_pandas()
print(df)

This will run the evaluation in synchronous mode and should avoid the RuntimeError related to the event loop [1][2].

Additionally, for the ExceptionInRunner issue, you can pass raise_exceptions=False in the evaluate function to show only a warning message instead of raising an exception:

result = evaluate(dataset, metrics=metrics, is_async=False, raise_exceptions=False)
df = result.to_pandas()
print(df)

This will prevent the runner thread from raising an exception and will only show a warning message instead [3].

To continue talking to Dosu, mention @dosu.

Elena4750 commented 1 month ago

Sad. Have tried, not worked.

LDelPinoNT commented 1 month ago

Getting a similar issue on Python 3.11 - latest version of RAGAS library, but when trying to generate a syntetic dataset with ".generate_with_langchain_docs".

Here is my traceback: RuntimeWarning: coroutine 'as_completed..sema_coro' was never awaited await self._event_pipe_gc() RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Rugved2204 commented 1 month ago

I am also getting a similar issue while generating a synthetic dataset.

Traceback: File "/home/ubuntu/.local/lib/python3.10/site-packages/ragas/executor.py", line 37, in sema_coro return await coro File "/home/ubuntu/.local/lib/python3.10/site-packages/ragas/executor.py", line 109, in wrapped_callable_async raise e File "/home/ubuntu/.local/lib/python3.10/site-packages/ragas/executor.py", line 104, in wrapped_callable_async result = await callable(*args, **kwargs) TypeError: object list can't be used in 'await' expression

LDelPinoNT commented 1 month ago

As additional info, I installed version 0.1.9 and the error disappeared.

tuantuan-lin commented 1 month ago

Hello, I have encountered the same situation, which has been successfully solved by the following methods.

https://github.com/explodinggradients/ragas/issues/1073

damlitos commented 2 weeks ago

Hello, I have encountered the same situation, which has been successfully solved by the following methods.

1073

what python version did you use?