openai / evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Other
14.97k stars 2.6k forks source link

pip install evals throws AssertionError #918

Open CholoTook opened 1 year ago

CholoTook commented 1 year ago

Describe the bug

$ pip install evals
...

python3 -m venv .venvSuccessfully built fire langdetect
ERROR: Exception:
Traceback (most recent call last):
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 389, in run
    to_install = resolver.get_installation_order(requirement_set)
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 188, in get_installation_order
    weights = get_topological_weights(
  File "/home/dan/Build/evals/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 276, in get_topological_weights
    assert len(weights) == expected_node_count
AssertionError

To Reproduce

python3 --version
# Python 3.10.6

python3 -m venv .venv
source .venv/bin/activate

pip --version
# pip 22.0.2 from /$PWD/.venv/lib/python3.10/site-packages/pip (python 3.10)

pip install evals
# Collecting evals
#  Downloading evals-1.0.3.post1-py3-none-any.whl (7.8 MB)
#     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 16.2 MB/s eta 0:00:00
# Collecting langdetect
# ...
# <as above>

Code snippets

Kernel: 5.15.0-71-generic x86_64 bits: 64 Desktop: Xfce 4.16.0
    Distro: Linux Mint 21.1 Vera

OS

Linux

Python version

Python 3.10.6

Library version

No idea....

CholoTook commented 1 year ago

This might be a fix:

pip install --upgrade pip
pip --version
# pip 23.1.2 from $PWD/.venv/lib/python3.10/site-packages/pip (python 3.10)
pip install evals
# Successfully installed evals-1.0.3.post1 snowflake-connector-python-3.0.3