search

truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2.07k stars 181 forks source link

"RuntimeError: cannot reuse already awaited coroutine" #859

Closed kouskouss closed 4 months ago

kouskouss commented 7 months ago

I am currently using the code provided in this Colab notebook to evaluate my RAG pipeline. While running the evaluation, I encountered an issue:

During the evaluation process, I encountered a RuntimeError with the message "cannot reuse already awaited coroutine" when executing the following code snippet:

with tru_rag as recording:
    for question in questions_500[:10]:
        answer = rag.query(question)

This error started occurring from yesterday, and I didn't have the same problems again. Is there any possible solution?

joshreini1 commented 7 months ago

Thanks for reporting @kouskouss - can you create a second issue and separate these? They are not related.

kouskouss commented 7 months ago

Thanks for reporting @kouskouss - can you create a second issue and separate these? They are not related.

Yes, of course.

joshreini1 commented 7 months ago

@piotrm0 can you take a look at this awaited coroutine issue?

piotrm0 commented 7 months ago

Having trouble replicating on main. @kouskouss can you let me know if the issue persists on the main branch of trulens_eval and the versions of pip packages and python you are using (python --version and pip list can list these)?

piotrm0 commented 7 months ago

Problem may be related to https://github.com/truera/trulens/issues/639 . Looking for a fix.

kouskouss commented 7 months ago

Having trouble replicating on main. @kouskouss can you let me know if the issue persists on the main branch of trulens_eval and the versions of pip packages and python you are using (python --version and pip list can list these)?

I am still face the same error : RuntimeError: cannot reuse already awaited coroutine when I run this:

with tru_rag as recording:
    for question in questions_500[:10]:
        answer = rag.query(question)
xcarlosamm commented 7 months ago

I have this issue too, downgrading the version doesn't solve the issue.

MARUD84 commented 7 months ago

I am also having the same issue ie RuntimeError: cannot reuse already awaited coroutine, even after downgrading to earlier version. Would be grateful if you can provide insights on how this can be addressed. Thanks.

vecorro commented 7 months ago

Hi,

I also see this issue when trying to replicate this notebook from the TruLens and LlamaIndex course from DeepLearning.ai using the latest versions of TruLens and LlamaIndex: Notebook. Of course I had to import LlamaIndex libraries and adjust the class methods call to accommodate to the changes coming with v0.10.5

Here the code snippet:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[7], line 10
      6 litellm.api_key = "NO_EVAL_KEY"
      8 tru_recorder = get_prebuilt_trulens_recorder(query_engine,
      9                                              app_id="Direct Query Engine")
---> 10 with tru_recorder as recording:
     11     for question in eval_questions:
     12         response = query_engine.query(question)

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/trulens_eval/app.py:842, in App.__exit__(self, exc_type, exc_value, exc_tb)
    839 self.recording_contexts.reset(ctx.token)
    841 if exc_type is not None:
--> 842     raise exc_value
    844 return

Cell In[7], line 12
     10 with tru_recorder as recording:
     11     for question in eval_questions:
---> 12         response = query_engine.query(question)
     13 records, feedback = tru.get_records_and_feedback(app_ids=[])
     14 records.head()

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/trulens_eval/instruments.py:633, in Instrument.tracked_method_wrapper.<locals>.tru_wrapper(*args, **kwargs)
    625 @functools.wraps(func)
    626 def tru_wrapper(*args, **kwargs):
    627     logger.debug(
    628         f"{query}: calling instrumented sync method {func} of type {type(func)}, "
    629         f"iscoroutinefunction={is_really_coroutinefunction(func)}, "
    630         f"isasyncgeneratorfunction={inspect.isasyncgenfunction(func)}"
    631     )
--> 633     return sync(tru_awrapper, *args, **kwargs)

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/trulens_eval/utils/asynchro.py:164, in sync(func, *args, **kwargs)
    162 # Get the return or error, return the return or raise the error.
    163 if thread.error is not None:
--> 164     raise thread.error
    165 else:
    166     return thread.ret

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/trulens_eval/utils/asynchro.py:152, in sync.<locals>.run_in_new_loop()
    150 try:
    151     loop = asyncio.new_event_loop()
--> 152     th.ret = loop.run_until_complete(awaitable)
    153 except Exception as e:
    154     th.error = e

File ~/miniconda3/envs/llm-env4/lib/python3.10/site-packages/nest_asyncio.py:98, in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File ~/miniconda3/envs/llm-env4/lib/python3.10/asyncio/futures.py:201, in Future.result(self)
    199 self.__log_traceback = False
    200 if self._exception is not None:
--> 201     raise self._exception.with_traceback(self._exception_tb)
    202 return self._result

File ~/miniconda3/envs/llm-env4/lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
    228 try:
    229     if exc is None:
    230         # We use the `send` method directly, because coroutines
    231         # don't have `__iter__` and `__next__` methods.
--> 232         result = coro.send(None)
    233     else:
    234         result = coro.throw(exc)
yuvneshtruera commented 6 months ago

@xcarlosamm @vecorro @MARUD84 @kouskouss Can you please let us know if this was fixed?

kouskouss commented 6 months ago

@yuvneshtruera Unfortunately, I still have the same issue "RuntimeError: cannot reuse already awaited coroutine"

joshreini1 commented 6 months ago

@kouskouss what version are you seeing this issue now?

yuvneshtruera commented 5 months ago

@kouskouss Can you please let us know the version?

kouskouss commented 5 months ago

@kouskouss Can you please let us know the version?

The problem still persists and my version is : Name: trulens-eval Version: 0.22.1

yuvneshtruera commented 5 months ago

@kouskouss Can you please update to the latest Trulens-eval version which is 0.28.2

joshreini1 commented 4 months ago

@kouskouss this issue is resolved by 0.28.2 - please let us know if you see otherwise.