truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2.01k stars 173 forks source link

Run 'tru.run_dashboard(port=8502)' Leaderboard shows error. #1040

Closed rajpr5656 closed 2 months ago

rajpr5656 commented 5 months ago

Bug Description Run 'tru.run_dashboard(port=8502)' Leaderboard shows error.

To Reproduce Not Sure on how to reproduce the steps as I am running standard Trulens Eval. I have enclosed screen shots as attachment and also all python packages. I am using python 3.12.2 version. Enclosed all Python packages below that are there. I have created a separate TruLens Leaderboard Issue.docx python environment and using there. However, I open Jupyter Lab in by generic folder. Package Version


aiohttp 3.9.3 aiosignal 1.3.1 alembic 1.13.1 altair 4.2.2 annotated-types 0.6.0 anyio 4.3.0 asgiref 3.8.1 asttokens 2.4.1 attrs 23.2.0 backoff 2.2.1 bcrypt 4.1.2 beautifulsoup4 4.12.3 blinker 1.7.0 bs4 0.0.2 build 1.2.1 cachetools 5.3.3 cargo 0.3 certifi 2024.2.2 charset-normalizer 3.3.2 chroma-hnswlib 0.7.3 chromadb 0.4.24 click 8.1.7 colorama 0.4.6 coloredlogs 15.0.1 comm 0.2.2 contourpy 1.2.0 cycler 0.12.1 dataclasses-json 0.6.4 debugpy 1.8.1 decorator 5.1.1 Deprecated 1.2.14 dill 0.3.8 dirtyjson 1.0.8 distro 1.9.0 entrypoints 0.4 executing 2.0.1 Faker 24.4.0 fastapi 0.110.0 favicon 0.7.0 filelock 3.13.3 flatbuffers 24.3.25 fonttools 4.50.0 frozendict 2.4.0 frozenlist 1.4.1 fsspec 2024.3.1 gitdb 4.0.11 GitPython 3.1.42 google-auth 2.29.0 googleapis-common-protos 1.63.0 greenlet 3.0.3 grpcio 1.62.1 h11 0.14.0 htbuilder 0.6.2 httpcore 1.0.5 httptools 0.6.1 httpx 0.27.0 huggingface-hub 0.22.1 humanfriendly 10.0 humanize 4.9.0 idna 3.6 importlib-metadata 6.11.0 importlib_resources 6.4.0 InstructorEmbedding 1.0.1 ipykernel 6.29.4 ipython 8.22.2 ipywidgets 8.1.2 jedi 0.19.1 Jinja2 3.1.3 joblib 1.3.2 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter_client 8.6.1 jupyter_core 5.7.2 jupyterlab_widgets 3.0.10 kiwisolver 1.4.5 kubernetes 29.0.0 langchain 0.1.13 langchain-community 0.0.29 langchain-core 0.1.35 langchain-text-splitters 0.0.1 langsmith 0.1.36 llama-index 0.10.25 llama-index-agent-openai 0.2.1 llama-index-cli 0.1.11 llama-index-core 0.10.25.post1 llama-index-embeddings-huggingface 0.2.0 llama-index-embeddings-instructor 0.1.3 llama-index-embeddings-openai 0.1.7 llama-index-indices-managed-llama-cloud 0.1.5 llama-index-legacy 0.9.48 llama-index-llms-openai 0.1.13 llama-index-multi-modal-llms-openai 0.1.4 llama-index-program-openai 0.1.5 llama-index-question-gen-openai 0.1.3 llama-index-readers-file 0.1.12 llama-index-readers-llama-parse 0.1.4 llama-parse 0.4.0 llamaindex-py-client 0.1.15 lxml 5.1.0 Mako 1.3.2 Markdown 3.6 markdown-it-py 3.0.0 markdownlit 0.0.7 MarkupSafe 2.1.5 marshmallow 3.21.1 matplotlib 3.8.3 matplotlib-inline 0.1.6 mdurl 0.1.2 merkle-json 1.0.0 millify 0.1.1 minijinja 1.0.16 mmh3 4.1.0 monotonic 1.6 more-itertools 10.2.0 mpmath 1.3.0 multidict 6.0.5 munch 4.0.0 mypy-extensions 1.0.0 nest-asyncio 1.6.0 networkx 3.2.1 nltk 3.8.1 numpy 1.26.4 oauthlib 3.2.2 onnxruntime 1.17.1 openai 1.14.3 opentelemetry-api 1.23.0 opentelemetry-exporter-otlp-proto-common 1.23.0 opentelemetry-exporter-otlp-proto-grpc 1.23.0 opentelemetry-instrumentation 0.44b0 opentelemetry-instrumentation-asgi 0.44b0 opentelemetry-instrumentation-fastapi 0.44b0 opentelemetry-proto 1.23.0 opentelemetry-sdk 1.23.0 opentelemetry-semantic-conventions 0.44b0 opentelemetry-util-http 0.44b0 orjson 3.10.0 overrides 7.7.0 packaging 23.2 pandas 2.2.1 parso 0.8.3 pillow 10.2.0 pip 24.0 platformdirs 4.2.0 posthog 3.5.0 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 4.25.3 psutil 5.9.8 pulsar-client 3.4.0 pure-eval 0.2.2 pyarrow 15.0.2 pyasn1 0.6.0 pyasn1_modules 0.4.0 pydantic 2.6.4 pydantic_core 2.16.3 pydeck 0.8.1b0 Pygments 2.17.2 pymdown-extensions 10.7.1 PyMuPDF 1.24.0 PyMuPDFb 1.24.0 pyparsing 3.1.2 pypdf 4.1.0 PyPika 0.48.9 pyproject_hooks 1.0.0 pyreadline3 3.4.1 python-dateutil 2.9.0.post0 python-decouple 3.8 python-dotenv 1.0.1 pytz 2024.1 pywin32 306 PyYAML 6.0.1 pyzmq 25.1.2 referencing 0.34.0 regex 2023.12.25 requests 2.31.0 requests-oauthlib 2.0.0 rich 13.7.1 rpds-py 0.18.0 rsa 4.9 safetensors 0.4.2 scikit-learn 1.4.1.post1 scipy 1.12.0 sentence-transformers 2.6.1 setuptools 69.2.0 six 1.16.0 smmap 5.0.1 sniffio 1.3.1 soupsieve 2.5 SQLAlchemy 2.0.29 st-annotated-text 4.0.1 stack-data 0.6.3 starlette 0.36.3 streamlit 1.32.2 streamlit-aggrid 1.0.0 streamlit-camera-input-live 0.2.0 streamlit-card 1.0.0 streamlit-embedcode 0.1.2 streamlit-extras 0.4.0 streamlit-faker 0.0.3 streamlit-image-coordinates 0.1.6 streamlit-keyup 0.2.3 streamlit-toggle-switch 1.0.2 streamlit-vertical-slider 2.5.5 striprtf 0.0.26 sympy 1.12 tenacity 8.2.3 threadpoolctl 3.4.0 tiktoken 0.6.0 tokenizers 0.15.2 toml 0.10.2 toolz 0.12.1 torch 2.2.2 tornado 6.4 tqdm 4.66.2 traitlets 5.14.2 transformers 4.39.1 trulens 0.13.4 trulens-eval 0.27.0 typer 0.11.0 typing_extensions 4.10.0 typing-inspect 0.9.0 tzdata 2024.1 urllib3 2.2.1 utils 1.0.2 uvicorn 0.29.0 watchdog 4.0.0 watchfiles 0.21.0 wcwidth 0.2.13 websocket-client 1.7.0 websockets 12.0 wheel 0.43.0 widgetsnbextension 4.0.10 wrapt 1.16.0 yarl 1.9.4 zipp 3.18.1

Expected behavior Expected to open Leaderboard and show results.

Relevant Logs/Tracebacks Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

Environment:

Additional context If you need logs let me know where to find.

TruLens Leaderboard Issue.docx

dosubot[bot] commented 5 months ago

Hey @rajpr5656! Good to see you tackling new challenges with TruLens. Let's dive into this dashboard dilemma together! 🚀

I understand that you're having trouble with the TruLens dashboard's Leaderboard not displaying as expected. Let's try to troubleshoot this issue step by step.

Firstly, it's important to ensure that all the packages are compatible with each other and with Python 3.12.2. Since Python 3.12 introduced several changes, some packages might not yet be fully compatible. You've mentioned aiohttp 3.9.3 and fastapi 0.110.0, which should be checked for compatibility with Python 3.12.2.

Secondly, although you're using TruLens version 0.13.4, it's a good practice to check if there's a newer version available that might have fixed the issue you're encountering. You can update TruLens to the latest version using the command pip install trulens --upgrade.

Next, ensure all dependencies are correctly installed and up to date. You can do this by running pip install -r requirements.txt if you have a requirements file or manually updating packages mentioned in your setup like aiohttp and fastapi using pip install aiohttp --upgrade and pip install fastapi --upgrade.

Also, ensure that the port 8502 is not already in use on your system. You can check this by using the netstat -aon | findstr 8502 command in the command prompt. If the port is in use, either close the application using it or choose a different port for the TruLens dashboard.

Sometimes, running the command prompt or PowerShell as an administrator can resolve issues related to permissions, especially on Windows 11. So, try running your application as an administrator.

Ensure that your firewall or antivirus is not blocking the application or the specific port you're trying to use. This can often cause unexpected issues.

When running tru.run_dashboard(port=8502), carefully check the console output for any errors or warnings that might give you more insight into what's going wrong.

If after following these steps the issue persists, it would be helpful to have more detailed information about the error messages you're receiving and the steps you've already taken to try to resolve the issue. This will help in diagnosing the problem more accurately.

For more information, you can refer to the TruLens codebase and documentation:

I hope this helps in resolving the issue. If you have any more questions or need further clarification, feel free to ask.

Sources

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
rajpr5656 commented 5 months ago

Thanks for your reply. Could you please let me know what version of Python should I reinstall to make tru.run_dashboard() to work. I will install recommended version of python. Also, pls recommend associated packages and its version to be installed. I will recreate all.

joshreini1 commented 4 months ago

@rajpr5656 can you share the stack trace?

rajpr5656 commented 4 months ago

Could you pls provide steps? I am sorry for asking basic question. If you are asking for sequence of operations of LLM then please find below: Followed below URL https://github.com/kevintsai/Building-and-Evaluating-Advanced-RAG-Applications/blob/main/L1-Advanced_RAG_Pipeline.ipynb

and only performed the following operations of

  1. 'Basic RAG pipeline'
  2. 'Evaluation setup using TruLens'
  3. Before running 'from utils import get_prebuilt_trulens_recorder

tru_recorder = get_prebuilt_trulens_recorder(query_engine, app_id="Direct Query Engine")' I copied your util.py code that has feedback codes.

  1. All ran good and I was able see results by running 'records.head()'. Issue is with 'tru.run_dashboard()'

Also, I degraded my python to 3.11.8 Here are my latest python packages: Package Version


aiohttp 3.9.3 aiosignal 1.3.1 aiostream 0.5.2 alembic 1.13.1 altair 4.2.2 annotated-types 0.6.0 anyio 4.3.0 asttokens 2.4.1 attrs 23.2.0 backoff 2.2.1 beautifulsoup4 4.12.3 bleach 6.1.0 blinker 1.7.0 cachetools 5.3.3 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 cohere 5.1.7 colorama 0.4.6 comm 0.2.2 contourpy 1.2.0 cycler 0.12.1 dataclasses-json 0.6.4 datasets 2.18.0 debugpy 1.8.1 decorator 5.1.1 Deprecated 1.2.14 dill 0.3.8 dirtyjson 1.0.8 distro 1.9.0 dnspython 2.6.1 entrypoints 0.4 executing 2.0.1 Faker 24.4.0 fastavro 1.9.4 favicon 0.7.0 filelock 3.13.3 fonttools 4.50.0 frozendict 2.4.0 frozenlist 1.4.1 fsspec 2024.2.0 gitdb 4.0.11 GitPython 3.1.42 greenlet 3.0.3 h11 0.14.0 htbuilder 0.6.2 html2text 2024.2.26 httpcore 1.0.5 httpx 0.27.0 huggingface-hub 0.22.2 humanize 4.9.0 idna 3.6 importlib_metadata 7.1.0 ipykernel 6.29.4 ipython 8.22.2 ipywidgets 8.1.2 jedi 0.19.1 Jinja2 3.1.3 joblib 1.3.2 jsonpatch 1.33 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter_client 8.6.1 jupyter_core 5.7.2 jupyterlab_widgets 3.0.10 kaggle 1.6.8 kiwisolver 1.4.5 langchain 0.1.13 langchain-community 0.0.29 langchain-core 0.1.36 langchain-text-splitters 0.0.1 langsmith 0.1.38 llama-index 0.10.25 llama-index-agent-openai 0.2.1 llama-index-cli 0.1.11 llama-index-core 0.10.25.post3 llama-index-embeddings-huggingface 0.2.0 llama-index-embeddings-openai 0.1.7 llama-index-indices-managed-llama-cloud 0.1.5 llama-index-legacy 0.9.48 llama-index-llms-openai 0.1.14 llama-index-multi-modal-llms-openai 0.1.4 llama-index-program-openai 0.1.5 llama-index-question-gen-openai 0.1.3 llama-index-readers-file 0.1.13 llama-index-readers-llama-parse 0.1.4 llama-parse 0.4.0 llamaindex-py-client 0.1.15 loguru 0.7.2 lxml 5.1.1 Mako 1.3.2 Markdown 3.6 markdown-it-py 3.0.0 markdownlit 0.0.7 MarkupSafe 2.1.5 marshmallow 3.21.1 matplotlib 3.8.3 matplotlib-inline 0.1.6 mdurl 0.1.2 merkle-json 1.0.0 millify 0.1.1 minijinja 1.0.16 more-itertools 10.2.0 mpmath 1.3.0 multidict 6.0.5 multiprocess 0.70.16 munch 4.0.0 mypy-extensions 1.0.0 nest-asyncio 1.6.0 networkx 3.2.1 nltk 3.8.1 numpy 1.26.4 openai 1.14.3 orjson 3.10.0 packaging 23.2 pandas 2.2.1 parso 0.8.3 pillow 10.2.0 pinecone-client 3.2.2 pip 24.0 platformdirs 4.2.0 prometheus_client 0.20.0 prompt-toolkit 3.0.43 protobuf 4.25.3 psutil 5.9.8 pure-eval 0.2.2 pyarrow 15.0.2 pyarrow-hotfix 0.6 pydantic 2.6.4 pydantic_core 2.16.3 pydeck 0.8.0 Pygments 2.17.2 pymdown-extensions 10.7.1 PyMuPDF 1.24.0 PyMuPDFb 1.24.0 pyparsing 3.1.2 pypdf 4.1.0 python-dateutil 2.9.0.post0 python-decouple 3.8 python-dotenv 1.0.1 python-slugify 8.0.4 pytz 2024.1 pywin32 306 PyYAML 6.0.1 pyzmq 25.1.2 referencing 0.34.0 regex 2023.12.25 requests 2.31.0 rich 13.7.1 rpds-py 0.18.0 safetensors 0.4.2 scikit-learn 1.4.1.post1 scipy 1.12.0 sentence-transformers 2.6.1 sentencepiece 0.2.0 setuptools 65.5.0 six 1.16.0 slack-bolt 1.18.1 slack_sdk 3.27.1 smmap 5.0.1 sniffio 1.3.1 soupsieve 2.5 SQLAlchemy 2.0.29 st-annotated-text 4.0.1 stack-data 0.6.3 streamlit 1.32.2 streamlit-aggrid 1.0.0 streamlit-camera-input-live 0.2.0 streamlit-card 1.0.0 streamlit-embedcode 0.1.2 streamlit-extras 0.4.0 streamlit-faker 0.0.3 streamlit-image-coordinates 0.1.6 streamlit-javascript 0.1.5 streamlit-keyup 0.2.3 streamlit-toggle-switch 1.0.2 streamlit-vertical-slider 2.5.5 striprtf 0.0.26 sympy 1.12 tenacity 8.2.3 text-unidecode 1.3 threadpoolctl 3.4.0 tiktoken 0.6.0 tokenizers 0.15.2 toml 0.10.2 toolz 0.12.1 torch 2.2.2 torchvision 0.17.2 tornado 6.4 tqdm 4.66.2 traitlets 5.14.2 transformers 4.39.2 trulens-eval 0.27.0 types-requests 2.31.0.20240311 typing_extensions 4.10.0 typing-inspect 0.9.0 tzdata 2024.1 tzlocal 5.2 urllib3 2.2.1 utils 1.0.2 validators 0.24.0 watchdog 4.0.0 wcwidth 0.2.13 webencodings 0.5.1 widgetsnbextension 4.0.10 win32-setctime 1.1.0 wrapt 1.16.0 xxhash 3.4.1 yarl 1.9.4 zipp 3.18.1

joshreini1 commented 4 months ago

I'm asking to copy the error you are seeing the leaderboard. Thanks

rajpr5656 commented 4 months ago

Hi Sorry for the delay. Please find below error message. I have also enclosed doc with screenshots. error screenshot.docx

STDERR Exception in thread ScriptRunner.scriptThread: Traceback (most recent call last): File "C:\Users\rajpr\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 1045, in _bootstrap_inner self.run() File "C:\Users\rajpr\AppData\Local\Programs\Python\Python311\Lib\threading.py", line 982, in run self._target(*self._args, **self._kwargs) File "C:\Users\rajpr\AppData\Local\Programs\Python\Python311\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 267, in _run_script_thread ctx = ScriptRunContext( ^^^^^^^^^^^^^^^^^ TypeError: ScriptRunContext.init() missing 1 required positional argument: 'main_script_path'

rajpr5656 commented 3 months ago

Hi Team, Do you have any updates please

joshreini1 commented 3 months ago

@rajpr5656 I'm unable to reproduce your error, however can you try:

tru.run_dashboard(force=True, port=8502)

joshreini1 commented 3 months ago

@rajpr5656 please give this a try, thanks!

sfc-gh-jreini commented 2 months ago

Closing due to inactivity