search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
5.65k stars 526 forks source link

Task was destroyed but it is pending! #664

Open pxiongw opened 4 months ago

pxiongw commented 4 months ago

Describe the bug I use ragas to evaluate my project,the dataset comes from the project, when I use 100 number of data,it is normal and could get the evaluate result.Then when I use more than 300 number of data,it raise exception as follow. I suspect there is something wrong with asyncio in ragas framework.

Exception ignored in: <coroutine object Executor.wrap_callable_with_index..wrapped_callable_async at 0x7fc4761f9310> Traceback (most recent call last): File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/metrics/_context_precision.py", line 129, in _ascore File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in call File "/home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/tenacity/init.py", line 334, in iter KeyError: 'idle_for' ........ Task was destroyed but it is pending! task: <Task pending name='context_recall-316' coro=<Executor.wrap_callable_with_index..wrapped_callable_async() done, defined at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py:90> wait_for= cb=[as_completed.._on_completion() at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/asyncio/tasks.py:558]> Task was destroyed but it is pending! task: <Task pending name='context_precision-316' coro=<Executor.wrap_callable_with_index..wrapped_callable_async() done, defined at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/site-packages/ragas/executor.py:90> wait_for= cb=[as_completed.._on_completion() at /home/gitlab-runner/anaconda3/envs/qa/lib/python3.10/asyncio/tasks.py:558]>

Ragas version: 0.1.2 Python version: 3.10.9

Code to Reproduce import nest_asyncio nest_asyncio.apply()

def run_evaluate(dataset,llm): result = evaluate( dataset=dataset, metrics=metrics, llm=llm, embeddings=build_embeddings(model_name), is_async=True, raise_exceptions=True, )

Error trace

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

weissenbacherpwc commented 4 months ago

same error here

nimbusaeta commented 4 months ago

Same here!

txrxso commented 3 months ago

Same here as well!

StephaneBereux commented 3 months ago

Same here :/

ChaocunChen commented 3 months ago

same

cryingjin commented 3 months ago

still same error....

sona-16 commented 2 months ago

I was trying to do "test generation" using RAGAS framework with the help of the "https://docs.ragas.io/en/stable/concepts/testset_generation.html", I'm facing error. Please have a look on the below error. ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead. Task was destroyed but it is pending! task: <Task pending name='Task-3' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit e-packages\ragas\executor.py:38> wait_for= cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]> Task was destroyed but it is pending! task: <Task pending name='Task-5' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit e-packages\ragas\executor.py:38> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]> Task was destroyed but it is pending! task: <Task pending name='Task-2' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\site-packages\ragas\executor.py:38> wait_for= cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>

JPonsa commented 2 months ago

Same here running enerator.generate_with_langchain_docs

from langchain_community.llms import VLLM

generator_llm = VLLM(model=args.generator,
                    trust_remote_code=True,  # mandatory for hf models
                    max_new_tokens=128,
                    top_k=10,
                    top_p=0.95,
                    temperature=0.8,
                    dtype="half",
                    enforce_eager=True
                            )

critic_llm = generator_llm.copy()

embeddings = HuggingFaceEmbeddings(model_name=args.embeddings)
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)

eval_ds = generator.generate_with_langchain_docs(
    docs,
    test_size=10
    distributions={
        simple: 0.4,
        reasoning: 0.4,
        multi_context: 0.3,
    },
)
eval_ds.to_pandas().to_csv(args.output)
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
        ^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py", line 605, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
           ^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompt=prompt, is_async=is_async)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 92, in generate
    return await agenerate_text_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
    output = await self._agenerate_helper(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper
    raise e
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper
    await self._agenerate(
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 544, in _agenerate
    return await run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/runnables/config.py", line 514, in run_in_executor
    return await asyncio.get_running_loop().run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_community/llms/vllm.py", line 132, in _generate
    outputs = self.client.generate(prompts, sampling_params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 190, in generate
    return self._run_engine(use_tqdm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 218, in _run_engine
    step_outputs = self.llm_engine.step()
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 683, in step
    return self._process_model_outputs(output, scheduler_outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 601, in _process_model_outputs
    self._process_sequence_group_outputs(seq_group, outputs)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 436, in _process_sequence_group_outputs
    parent_child_dict[sample.parent_seq_id].append(sample)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 139
Traceback (most recent call last):
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/./src/evaluation/RAGAS.py", line 205, in <module>
    main(args, True)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/./src/evaluation/RAGAS.py", line 111, in main
    eval_ds = generator.generate_with_langchain_docs(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/generator.py", line 175, in generate_with_langchain_docs
    self.docstore.add_documents(
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/docstore.py", line 215, in add_documents
    self.add_nodes(nodes, show_progress=show_progress)
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/docstore.py", line 254, in add_nodes
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.

Processed prompts: 2it [00:07,  3.56s/it]                       
Processed prompts: 3it [00:07,  2.41s/it]

Processed prompts:   0%|          | 0/3 [00:05<?, ?it/s]
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x2acc911eee40>
Traceback (most recent call last):
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/testset/extractor.py", line 49, in extract
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 92, in generate
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 47, in __call__
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 449, in result
  File "/shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/tenacity/_asyncio.py", line 50, in __call__
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/llms/base.py", line 177, in agenerate_text
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
  File "/lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
RuntimeError: coroutine ignored GeneratorExit
Task was destroyed but it is pending!
task: <Task pending name='Task-8' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-22' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-10' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-24' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-12' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-26' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-14' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
Task was destroyed but it is pending!
task: <Task pending name='Task-6' coro=<as_completed.<locals>.sema_coro() running at /lustre/scratch/scratch/<user_id>/ctgov_rag/.venv/lib/python3.11/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/futures.py:387, Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /shared/ucl/apps/python/3.11.3/gnu-4.9.2/lib/python3.11/asyncio/tasks.py:592]>
laura-ham commented 2 months ago

I experienced the same issue. After some time of debugging, I found that leaving out this import line of another module that I used elsewhere in the code solved the issue: from test_app import run. No idea how and why this import line caused this error you all describe above.

jasonngap1 commented 1 month ago

Experienced the same problem as well.

jmgu0212 commented 2 weeks ago

Same error here!

GarrickLin commented 2 weeks ago

Same error here!

changwangss commented 1 week ago

same error