Error Executing 'llama_index_quickstart.py' and Integrating with RAG: Common Issue Across Examples.

[shreyashpatel-crest]

I encountered an execution error while running the "llama_index_tru_test.py" from the examples directory. The same issue arose during an earlier attempt to integrate trueval_lens with my RAG (Retrieval-Augmented Generation) code. Below is the traceback of the error encountered

Traceback (most recent call last):
  File "/home/devuser/RAG_chatbot/llama_index_tru_test.py", line 108, in <module>
    tru_query_engine_recorder = TruLlama(
                                ^^^^^^^^^
  File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/tru_llama.py", line 245, in __init__
    kwargs['instrument'] = LlamaInstrument(app=self)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/tru_llama.py", line 178, in __init__
    include_classes=LlamaInstrument.Default.CLASSES(),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/tru_llama.py", line 86, in <lambda>
    BaseComponent,
    ^^^^^^^^^^^^^
NameError: name 'BaseComponent' is not defined

Environment Python version: 3.11 Operating System: CentOS7

Would appreciate any guidance on resolving this issue or if there's a workaround available.

[aaronvarghese]

Can you verify whether your environment has llama_index version 0.9.15post2 or newer installed?

[shreyashpatel-crest]

I need to check on that.

[aaronvarghese]

If llama_index is installed, it's possible that the issue you're encountering is connected to the one described here: https://github.com/truera/trulens/issues/898

[shreyashpatel-crest]

Hey, @aaronvarghese. I encountered the NameError: name 'BaseComponent' is not defined error while working with version 0.9.13 of llama-index. After that, I have tested the code with two llama-index versions. Let me share that with you.

1. For llama-index==0.9.15post2 version I'm facing the below error:

   
   Traceback (most recent call last):
   File "/home/devuser/RAG_chatbot/llama_index_tru_test.py", line 130, in <module>
   display(rec)
   ^^^^^^^
   NameError: name 'display' is not defined
   Feedback Function exception caught: Traceback (most recent call last):
   File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 112, in sync
   loop = asyncio.get_running_loop()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
   RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 550, in run result_and_meta, part_cost = sync( ^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 117, in sync return loop.run_until_complete(awaitable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete return f.result() ^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/futures.py", line 203, in result raise self._exception.with_traceback(self._exception_tb) File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/tasks.py", line 277, in step result = coro.send(None) ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 458, in atrack_all_costs_tally result, cbs = await Endpoint.atrack_all_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 439, in atrack_all_costs return await Endpoint._atrack_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 536, in _atrack_costs result: T = await desync(func, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 88, in desync res = await asyncio.to_thread(func, args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/threads.py", line 25, in to_thread return await loop.run_in_executor(None, func_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor executor.submit(func, args), loop=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/threading.py", line 75, in submit return super().submit( ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 555, in run raise RuntimeError( RuntimeError: Evaluation of groundedness_measure_with_cot_reasons failed on inputs: {'source': ['|\n' '![](https://sep.turbifycdn.com/ca/Img/tr cannot schedule new futures after interpreter shutdown.

Feedback Function exception caught: Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 112, in sync loop = asyncio.get_running_loop() ^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 550, in run result_and_meta, part_cost = sync( ^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 117, in sync return loop.run_until_complete(awaitable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete return f.result() ^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/futures.py", line 203, in result raise self._exception.with_traceback(self._exception_tb) File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/tasks.py", line 277, in step result = coro.send(None) ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 458, in atrack_all_costs_tally result, cbs = await Endpoint.atrack_all_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 439, in atrack_all_costs return await Endpoint._atrack_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 536, in _atrack_costs result: T = await desync(func, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 88, in desync res = await asyncio.to_thread(func, args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/threads.py", line 25, in to_thread return await loop.run_in_executor(None, func_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor executor.submit(func, args), loop=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/threading.py", line 75, in submit return super().submit( ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 555, in run raise RuntimeError( RuntimeError: Evaluation of relevance failed on inputs: {'prompt': 'What did the author do growing up?', 'response': 'The author mentioned that before college, they worked on two '

cannot schedule new futures after interpreter shutdown.

Feedback Function exception caught: Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 112, in sync loop = asyncio.get_running_loop() ^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 550, in run result_and_meta, part_cost = sync( ^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 117, in sync return loop.run_until_complete(awaitable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete return f.result() ^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/futures.py", line 203, in result raise self._exception.with_traceback(self._exception_tb) File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/tasks.py", line 277, in step result = coro.send(None) ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 458, in atrack_all_costs_tally result, cbs = await Endpoint.atrack_all_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 439, in atrack_all_costs return await Endpoint._atrack_costs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 536, in _atrack_costs result: T = await desync(func, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/asynchro.py", line 88, in desync res = await asyncio.to_thread(func, args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/threads.py", line 25, in to_thread return await loop.run_in_executor(None, func_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/asyncio/base_events.py", line 829, in run_in_executor executor.submit(func, args), loop=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/utils/threading.py", line 75, in submit return super().submit( ^^^^^^^^^^^^^^^ File "/home/devuser/anaconda3/envs/py311/lib/python3.11/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/devuser/anaconda3/envs/py311/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 555, in run raise RuntimeError( RuntimeError: Evaluation of qs_relevance failed on inputs: {'question': 'What did the author do growing up?', 'statement': '|\n'

cannot schedule new futures after interpreter shutdown.

2. For latest version `llama-index==0.10.5` version I'm facing the below error:

Traceback (most recent call last): File "/home/devuser/RAG_chatbot/llama_index_tru_test.py", line 47, in from llama_index.readers.web import SimpleWebPageReader ModuleNotFoundError: No module named 'llama_index.readers.web' (py311_1) [root@co7-50-7-133 RAG_chatbot]# python3 llama_index_tru_test.py 🦑 Tru initialized with db url sqlite:///default.sqlite . 🛑 Secret keys may be written to the database. See the database_redact_keys option of Tru to prevent this. Traceback (most recent call last): File "/home/devuser/RAG_chatbot/llama_index_tru_test.py", line 47, in from llama_index.readers.web import SimpleWebPageReader ModuleNotFoundError: No module named 'llama_index.readers.web'

I checked llama-index documentation also for how to use `SimpleWebPageReader`. It seems correct to me.

[joshreini1]

Hey @shreyashpatel-crest - support for Llama-Index versions > 0.10 coming in #891 , can you install from that branch and give it a try? Thanks!

[shreyashpatel-crest]

Hey @joshreini1, Cool. Will try that solution and let you know.

[joshreini1]

Support for Llama-Index > 0.10 is merged to main and will be released in 0.24 today or tomorrow.

@shreyashpatel-crest please let me know if you have any persistent issues. Thanks!

[joshreini1]

Support for Llama-Index > 0.10 now released in 0.24.0!