gkamradt / LLMTest_NeedleInAHaystack

Doing simple retrieval from LLM models at various context lengths to measure accuracy
Other
1.28k stars 132 forks source link

does it run at all? Basic commands failed to run as per the README. #43

Open iitrsamrat opened 3 months ago

iitrsamrat commented 3 months ago

steps to reproduce environment Conda python 3.9

pip install needlehaystack ` needlehaystack.run_test --provider anthropic --model_name "claude-2.1" --document_depth_percents "[50]" --context_lengths "[2000]" /Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/langchain/callbacks/init.py:37: LangChainDeprecationWarning: Importing this callback from langchain is deprecated. Importing it from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

from langchain_community.callbacks import base.

To install langchain-community run pip install -U langchain-community. warnings.warn( Traceback (most recent call last): File "/Users/samsaha2/miniconda3/envs/nih/bin/needlehaystack.run_test", line 5, in from needlehaystack.run import main File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/init.py", line 1, in from .llm_needle_haystack_tester import LLMNeedleHaystackTester File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/llm_needle_haystack_tester.py", line 10, in from .providers import ModelProvider File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/providers/init.py", line 1, in from .anthropic import Anthropic File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/providers/anthropic.py", line 12, in from .model import ModelProvider File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/providers/model.py", line 5, in class ModelProvider(ABC): File "/Users/samsaha2/miniconda3/envs/nih/lib/python3.9/site-packages/needlehaystack/providers/model.py", line 10, in ModelProvider def generate_prompt(self, context: str, retrieval_question: str) -> str | list[dict[str, str]]: ... TypeError: unsupported operand type(s) for |: 'type' and 'types.GenericAlias'

`

` running pip install -U langchain-community create conflict

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. needlehaystack 0.1.0 requires langchain-core==0.1.26, but you have langchain-core 0.1.36 which is incompatible.

`

iitrsamrat commented 3 months ago

` with python 3.12.1. 3.12.2 & 3.10 failed to install requirements

Building wheels for collected packages: multidict Building wheel for multidict (pyproject.toml) ... error error: subprocess-exited-with-error

× Building wheel for multidict (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [200 lines of output]


`