[Bug]: can't run tests with `pytest` from repo root

[jamesbraza]

What happened?

This should work:

git clone git@github.com:BerriAI/litellm.git
cd litellm
python -m venv venv && source venv/bin/activate
pip install poetry
poetry install --with dev
pytest tests # Fails
pytest litellm/tests # Fails

Second to last line, I get the below error:

Test session starts (platform: darwin, Python 3.11.8, pytest 8.1.1, pytest-sugar 1.0.0)
rootdir: /Users/jamesbraza/code/litellm
configfile: pyproject.toml
plugins: sugar-1.0.0, anyio-4.3.0, time-machine-2.14.0, timer-1.0.0, asyncio-0.23.5.post1
asyncio: mode=Mode.STRICT
collecting ...
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_keys.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_keys.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_keys.py:15: in <module>
    from litellm.proxy._types import LitellmUserRoles
E   ModuleNotFoundError: No module named 'litellm.proxy'

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_ratelimit.py ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_ratelimit.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_ratelimit.py:17: in <module>
    from litellm import utils, Router
E   ImportError: cannot import name 'utils' from 'litellm' (unknown location)

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ERROR collecting tests/test_whisper.py ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
ImportError while importing test module '/path/to/litellm/tests/test_whisper.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.11.8/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_whisper.py:26: in <module>
    from litellm import Router
E   ImportError: cannot import name 'Router' from 'litellm' (unknown location)

Last line, I get the below error:

ImportError while loading conftest '/path/to/litellm/litellm/tests/conftest.py'.
litellm/tests/conftest.py:12: in <module>
    import litellm
E   ModuleNotFoundError: No module named 'litellm'

Can we fix the Poetry --> pytest workflow

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

@jamesbraza what needs to be changed here?

cd litellm/tests
pytest . -x -vv

works

cd litellm/litellm/tests
pytest . -x -vv

works

[jamesbraza]

So check this:

git clone git@github.com:BerriAI/litellm.git
cd litellm
python -m venv venv && source venv/bin/activate
pip install poetry
poetry install --with dev
pytest tests # Fails with ImportError: cannot import name 'Router' from 'litellm' (unknown location)
cd tests
pytest # Fails with ModuleNotFoundError: No module named 'openai.types.beta.threads.message_content'
cd ../litellm/tests
pytest # Fails with ModuleNotFoundError: No module named 'openai.types.beta.threads.message_content'
pytest . # Fails with ModuleNotFoundError: No module named 'fastapi', 'opentelemetry', 'backoff'

I am more just sharing that LiteLLM is somewhat sensitive to starting directory because pytest only works when run from specific directories with specific parameterizations (e.g. specifying a file_or_dir of .).

If one sets things up nicely (e.g. specifying testpaths in a config), one should just be able to invoke pytest from the repo root and have everything work.

One other issue is that poetry install --with dev doesn't seem to install everything necessary to run tests. Maybe there should be more Poetry groups made, or the dev group should be expanded

---

Also, feel free to close this out. I am more just pointing out the "getting started" developing has opportunities for improvement