Given code @ https://github.com/explodinggradients/ragas is not working and raises an exeption.

I can not figure out what is wrong with the provided code read the documentation provided as well.

Bug ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead.

Ragas version: Python version:

Code to Reproduce pip install ragas

from datasets import Dataset 
import os
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
    'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
    'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
    'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
    ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
    'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
score.to_pandas()

Error trace

Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete
    self._check_running()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:235: RuntimeWarning: coroutine 'as_completed.<locals>.sema_coro' was never awaited
  results = executor.results()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-5-58c56f78de7c>](https://localhost:8080/#) in <cell line: 1>()
----> 1 score = evaluate(dataset,metrics=[faithfulness,answer_correctness])
      2 score.to_pandas()

1 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
    235         results = executor.results()
    236         if results == []:
--> 237             raise ExceptionInRunner()
    238 
    239         # convert results to dataset_like
**Expected behavior**
code should run and evaluate the given responses

Additional context Tried to run as it is code provided in the repo but it is raising an error. tried it on 03-JUL-2024

Hey @muhammad-ammar12! 👋 I'm here to help you with your coding troubles!

The "ExceptionInRunner" error you're encountering is due to the event loop already running when the evaluate function tries to execute asynchronous operations. This is a common issue when running asynchronous code in environments like Jupyter notebooks.

To fix this, you can use the nest_asyncio library to allow nested use of asyncio.run(). Here's how you can modify your code:

Install the nest_asyncio library:
```
pip install nest_asyncio
```

Modify your code to include nest_asyncio:

from datasets import Dataset 
import os
import nest_asyncio
from ragas import evaluate
from ragas.metrics import faithfulness, answer_correctness

# Apply nest_asyncio to allow nested event loops
nest_asyncio.apply()

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = {
   'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
   'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
   'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
   ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
   'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset, metrics=[faithfulness, answer_correctness])
score.to_pandas()

This should resolve the "This event loop is already running" error and allow your code to run correctly ^[1].

To continue talking to Dosu, mention @dosu.

thanks for reaching out, tried your provided solution but still getting the same error.

On Wed, Jul 3, 2024 at 4:16 PM dosubot[bot] @.***> wrote:

Hey @muhammad-ammar12 https://github.com/muhammad-ammar12! 👋 I'm here to help you with your coding troubles!

The "ExceptionInRunner" error you're encountering is due to the event loop already running when the evaluate function tries to execute asynchronous operations. This is a common issue when running asynchronous code in environments like Jupyter notebooks.

To fix this, you can use the nest_asyncio library to allow nested use of asyncio.run(). Here's how you can modify your code:

1.

Install the nest_asyncio library:

pip install nest_asyncio

2.

Modify your code to include nest_asyncio:

from datasets import Dataset import osimport nest_asynciofrom ragas import evaluatefrom ragas.metrics import faithfulness, answer_correctness

Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()

os.environ["OPENAI_API_KEY"] = "your-openai-key" data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset, metrics=[faithfulness, answer_correctness])score.to_pandas()

This should resolve the "This event loop is already running" error and allow your code to run correctly [1] https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py .

To continue talking to Dosu https://dosu.dev, mention @dosu https://github.com/dosu.

— Reply to this email directly, view it on GitHub https://github.com/explodinggradients/ragas/issues/1064#issuecomment-2206505060, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7ROIEUQVCZJRQEL47XKBRDZKQIVTAVCNFSM6AAAAABKJZJRO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGUYDKMBWGA . You are receiving this because you were mentioned.Message ID: @.***>

My Bad! just figured out after fixing the code with your help was missing the API_key. it works now!!

thanks a lot for your help @dosu

On Wed, Jul 3, 2024 at 4:39 PM Muhammad Ammar Shahid < @.***> wrote:

thanks for reaching out, tried your provided solution but still getting the same error.

On Wed, Jul 3, 2024 at 4:16 PM dosubot[bot] @.***> wrote:

Hey @muhammad-ammar12 https://github.com/muhammad-ammar12! 👋 I'm here to help you with your coding troubles!

The "ExceptionInRunner" error you're encountering is due to the event loop already running when the evaluate function tries to execute asynchronous operations. This is a common issue when running asynchronous code in environments like Jupyter notebooks.

To fix this, you can use the nest_asyncio library to allow nested use of asyncio.run(). Here's how you can modify your code:

1.

Install the nest_asyncio library:

pip install nest_asyncio

2.

Modify your code to include nest_asyncio:

from datasets import Dataset import osimport nest_asynciofrom ragas import evaluatefrom ragas.metrics import faithfulness, answer_correctness

Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()

os.environ["OPENAI_API_KEY"] = "your-openai-key" data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset, metrics=[faithfulness, answer_correctness])score.to_pandas()

This should resolve the "This event loop is already running" error and allow your code to run correctly [1] https://github.com/explodinggradients/ragas/blob/main/src/ragas/executor.py .

To continue talking to Dosu https://dosu.dev, mention @dosu https://github.com/dosu.

— Reply to this email directly, view it on GitHub https://github.com/explodinggradients/ragas/issues/1064#issuecomment-2206505060, or unsubscribe https://github.com/notifications/unsubscribe-auth/A7ROIEUQVCZJRQEL47XKBRDZKQIVTAVCNFSM6AAAAABKJZJRO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGUYDKMBWGA . You are receiving this because you were mentioned.Message ID: @.***>

You're welcome! I'm glad to hear it's working now. If you have any more questions or run into any other issues, feel free to ask. Happy coding! 🚀

To continue talking to Dosu, mention @dosu.

explodinggradients / ragas

Given code @ https://github.com/explodinggradients/ragas is not working and raises an exeption. #1064

Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()

Apply nest_asyncio to allow nested event loopsnest_asyncio.apply()