ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

[edubrigham]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug Can't get evaluation to work. constantly get the error: The runner thread which was running the jobs raised an exception

Ragas version: latest Python version: 3

Code to Reproduce

from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
)

result = evaluate(
    dataset,
    metrics=[

        faithfulness,
        answer_relevancy,

    ],
)

result

Error trace

Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete
    self._check_running()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
/usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:235: RuntimeWarning: coroutine 'as_completed.<locals>.sema_coro' was never awaited
  results = executor.results()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
[<ipython-input-12-5dadbbc00d9a>](https://localhost:8080/#) in <cell line: 3>()
      1 from ragas import evaluate
      2 
----> 3 result = evaluate(
      4     dataset,
      5     metrics=[

1 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
    235         results = executor.results()
    236         if results == []:
--> 237             raise ExceptionInRunner()
    238 
    239         # convert results to dataset_like

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

[dosubot[bot]]

Hey there, @edubrigham! I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this together!

To resolve the "The runner thread which was running the jobs raised an exception" error, you can pass the raise_exceptions=False parameter to the evaluate function. This will show only a warning message instead of raising an exception, allowing the execution to continue.

Here's how you can modify your code:

from ragas import evaluate
from ragas.metrics import answer_relevancy, faithfulness

result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy],
    raise_exceptions=False  # Add this line
)

print(result)

This should prevent the runner thread from raising an exception and will only show a warning message instead ^[1][2].

To continue talking to Dosu, mention @dosu.

[Islam231bi]

I have the same problem but with synthetic test case generation

import os

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_cohere.llms import Cohere    
from langchain_cohere import CohereEmbeddings

os.environ["COHERE_API_KEY"] = "***************************"

# generator with openai models
generator_llm = Cohere(model="command", temperature=0.75)
critic_llm = Cohere(model="command", temperature=0.75)
embeddings = CohereEmbeddings(model="embed-english-light-v3.0")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
print(type(testset))

testset.to_pandas()

Error: Exception in thread Thread-14: Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 87, in run results = self.loop.run_until_complete(self._aresults()) File "/usr/lib/python3.10/asyncio/base_events.py", line 625, in run_until_complete self._check_running() File "/usr/lib/python3.10/asyncio/base_events.py", line 584, in _check_running raise RuntimeError('This event loop is already running') RuntimeError: This event loop is already running /usr/lib/python3.10/threading.py:1018: RuntimeWarning: coroutine 'Runner._aresults' was never awaited self._invoke_excepthook(self) RuntimeWarning: Enable tracemalloc to get the object allocation traceback /usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py:252: RuntimeWarning: coroutine 'as_completed..sema_coro' was never awaited results = executor.results() RuntimeWarning: Enable tracemalloc to get the object allocation traceback

ExceptionInRunner Traceback (most recent call last) in <cell line: 22>() 20 21 # generate testset ---> 22 testset = generator.generate_with_langchain_docs(documents, raise_exceptions=False, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25}) 23 print(type(testset)) 24

2 frames /usr/local/lib/python3.10/dist-packages/ragas/testset/docstore.py in add_nodes(self, nodes, show_progress) 252 results = executor.results() 253 if not results: --> 254 raise ExceptionInRunner() 255 256 for i, n in enumerate(nodes):

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead.

[ZG3Z]

I have the same problem with synthetic test generation

[aixelbsck]

Hello, i had similar issue and handle it by adding the following code before my ragas calls :

pip install nest-asyncio

and

import nest_asyncio nest_asyncio.apply()

Hope it helps.

[anujmehta]

@aixelbsck I tried using nest_asyncio but still getting same error. I was trying to execute following https://docs.ragas.io/en/stable/getstarted/testset_generation.html on Mac as a jupyter notebook and getting following error

Cell In[26], [line 3](vscode-notebook-cell:?execution_count=26&line=3)
      [1](vscode-notebook-cell:?execution_count=26&line=1) nest_asyncio.apply()
      [2](vscode-notebook-cell:?execution_count=26&line=2) generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
----> [3](vscode-notebook-cell:?execution_count=26&line=3) testset = generator.generate_with_langchain_docs(documents, test_size=2, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

File ~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:206, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    [204](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:204) distributions = distributions or {}
    [205](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:205) # chunk documents and add to docstore
--> [206](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:206) self.docstore.add_documents(
    [207](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:207)     [Document.from_langchain_document(doc) for doc in documents]
    [208](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:208) )
    [210](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:210) return self.generate(
    [211](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:211)     test_size=test_size,
    [212](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:212)     distributions=distributions,
   (...)
    [216](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:216)     run_config=run_config,
    [217](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/generator.py:217) )

File ~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:215, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    [210](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:210) # split documents with self.splitter into smaller nodes
    [211](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:211) nodes = [
    [212](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:212)     Node.from_langchain_document(d)
    [213](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:213)     for d in self.splitter.transform_documents(docs)
...
--> [254](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:254)     raise ExceptionInRunner()
    [256](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:256) for i, n in enumerate(nodes):
    [257](https://file+.vscode-resource.vscode-cdn.net/Users/anujmehta/Python%20Workspace/ragas-demo/ragas_demo/~/Library/Caches/pypoetry/virtualenvs/ragas-demo-53iINUe4-py3.11/lib/python3.11/site-packages/ragas/testset/docstore.py:257)     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?5ef1a4b6-b3cc-400f-8753-824b83dd5465) or open in a [text editor](command:workbench.action.openLargeOutput?5ef1a4b6-b3cc-400f-8753-824b83dd5465). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...