charmed-kubernetes / pytest-operator

Apache License 2.0
6 stars 13 forks source link

version 0.14.0 causes intermittent problems with pytest-asyncio's `--asyncio-mode` command line arg #65

Closed ca-scribner closed 2 years ago

ca-scribner commented 2 years ago

Typically, we expect that pip install pytest-operator will also install pytest-asyncio, and pytest-asyncio adds a command line argument to pytest for --asyncio-mode (eg: pytest --asyncio-mode=auto should be a valid command). However, after commit cfef8ea9927e4cc07ff69d042239e2612d1cfc84, this command sometimes works. For example, running the following in an empty dir:

# pip install direct from commit "Adding option --no-deploy" (feature commit 2)
# BREAKS
for x in `seq 20 29`; do

    python -m venv venv$x; source venv$x/bin/activate; python -m pip install -U pip -q
    pip install pytest git+https://github.com/charmed-kubernetes/pytest-operator.git@cfef8ea9927e4cc07ff69d042239e2612d1cfc84 -q
    ./venv$x/bin/pytest --asyncio-mode=auto  # <-- this should execute successfully

done

Will result in 10 new venvs, some of which can successfully execute the pytest --asyncio-mode=auto command and some of which give the following error:

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --asyncio-mode=auto
  inifile: None
  rootdir: /tmp/pytest-asyncio-debug

This intermittent error is linked to the venv, not to execution of the pytest command. For example, if you create a venv and then execute pytest --asyncio-mode=auto multiple times, all executions will have the same result (either pass or fail). But if you then create multiple venvs and repeat this test, some venvs will pass and some will fail.

An example of installs that all work can be generated by using the previous commit:

# pip install direct from commit "Replace build ..." (feature commit 1)
# works
for x in `seq 11 20`; do

    python -m venv venv$x; source venv$x/bin/activate; python -m pip install -U pip -q
    pip install pytest git+https://github.com/charmed-kubernetes/pytest-operator.git@182f46d264c8e3ff94e74a0f863f94a4e269e254 -q
    ./venv$x/bin/pytest --asyncio-mode=auto

done

Workarounds

Specifying the asyncio_mode via a pytest.ini or pyproject.toml appears to work consistently. For example, in pyproject.toml this always works:

[tool.pytest.ini_options]
asyncio_mode = "auto"
ca-scribner commented 2 years ago

from @r-f-g, the problem appears to be traceable to these lines of code. Removing them seems to fix the issue

ca-scribner commented 2 years ago

It appears that pip install -e my/local/version always results in a working venv, and that building a whl of pytest-operator locally then installing that whl (python setup.py sdist bdist_wheel; pip install ./dist/pytest_operator-0.14.0-py3-none-any.whl) fails at least sometimes (2 failures in 2 attempts)