ionelmc
commented
3 years ago Some info about what OS you have could be useful.
Open wpoxon opened 3 years ago
ionelmc
commented
3 years ago Some info about what OS you have could be useful.
wpoxon
commented
3 years ago Sorry, forgot that part...
wpoxon@dev-2:~/src/pytest-benchmark$ uname -a
Linux dev-2 5.4.0-1035-aws #37-Ubuntu SMP Wed Jan 6 21:01:57 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxAnd here are the steps to reproduce the issue I am seeing on the system above:
cd
sudo apt-get install python3-venv # venv not installed by default, we need to install it
python3 -m venv .toxbase
.toxbase/bin/pip list # not much, but we have have pip!
.toxbase/bin/pip install tox
.toxbase/bin/tox --version
mkdir ~/bin # edit your $PATH in ~/.profile or ~/.bash_profile to include ${HOME}/bin
ln -s ~/.toxbase/bin/tox ~/bin/tox
which tox # tells me ${HOME}/bin/tox
tox --version # works
PY_VERSIONS="2.7.17 3.5.10 3.6.12 3.7.9 3.8.6 pypy-5.7.1 pypy3.6-7.3.1"
for ver in ${PY_VERSIONS}
do
pyenv uninstall -f ${ver}
pyenv install -f ${ver}
done
pyenv global ${PY_VERSIONS}
pyenv versions
cd ~src/pytest-benchmark
rm -rf .tox
toxWhat appears to be happening is that sometime after the first successful execution of tests in an environment that includes cover (py27-pytest46-pygal24-nodist-cover), the {toxinidir}/.coverage file ends up getting set to zero length.
This is causing runs in any later environments that include cover to fail (py27-pytest46-pygal24-xdist-cover, pypy-pytest46-pygal24-nodist-cover, ...). I will try to narrow down when and why the {toxinidir}/.coverage file is getting set to zero length.
wpoxon
commented
3 years ago To help track this down, I created a fixture that runs code before and after each test. The fixture checks if {toxinidir}/.coverage exists. If the file exists and its size is 0, the fixture will assert.
I structured the fixture this way because If {toxinidir}/.coverage does not exist, there is no problem. If {toxinidir}/.coverage exists and has valid content, there is no problem. If {toxinidir}/.coverage exists and has invalid content (when I see this it is because the file is empty / size == 0), then subsequent coverage tests will fail.
I re-ran tox with this fixture installed and narrowed the failure down to this test:
py27-pytest46-pygal24-xdist-cover start: run-test
py27-pytest46-pygal24-xdist-cover run-test: commands[0] | py.test --cov=src --cov-report=term-missing --cov-append -vv
setting PATH=/home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin:/home/users/wpoxon/.pyenv/plugins/pyenv-virtualenv/shims:/home/users/wpoxon/.pyenv/shims:/home/users/wpoxon/.pyenv/bin:/home/users/wpoxon/.pyenv/bin:/home/users/wpoxon/.local/bin:/home/users/wpoxon/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[953602] /home/users/wpoxon/src/pytest-benchmark$ /home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin/py.test --cov=src --cov-report=term-missing --cov-append -vv
============================================================ test session starts ============================================================
platform linux2 -- Python 2.7.17, pytest-4.6.11, py-1.10.0, pluggy-0.13.1 -- /home/users/wpoxon/src/pytest-benchmark/.tox/py27-pytest46-pygal24-xdist-cover/bin/python
cachedir: .tox/py27-pytest46-pygal24-xdist-cover/.pytest_cache
benchmark: 3.2.3 (defaults: timer=time.time disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/users/wpoxon/src/pytest-benchmark, inifile: setup.cfg, testpaths: tests
plugins: aspectlib-1.5.1, xdist-1.34.0, instafail-0.4.2, travis-fold-1.3.0, forked-1.3.0, cov-2.11.1, benchmark-3.2.3
collected 231 items
tests/test_benchmark.py::test_help PASSED [ 0%]
tests/test_benchmark.py::test_groups PASSED [ 0%]
tests/test_benchmark.py::test_group_by_name PASSED [ 1%]
...
tests/test_benchmark.py::test_bad_rounds PASSED [ 8%]
tests/test_benchmark.py::test_bad_rounds_2 PASSED [ 9%]
tests/test_benchmark.py::test_compare PASSED [ 9%]
tests/test_benchmark.py::test_compare ERROR [ 9%]So what I know at this point is: When tests/test_benchmark.py::test_compare begins running within the py27-pytest46-pygal24-xdist-cover environment, {toxinidir}/.coverage exists and has a size > 0. When tests/test_benchmark.py::test_compare is done running within the py27-pytest46-pygal24-xdist-cover environment, {toxinidir}/.coverage still exists and its size is now 0 which sets up for us to begin hitting the failures reported in this issue as coverage tests in this and other environments run.
john-science
commented
2 years ago Was there ever a resolution to this issue? I'm having what I believe is the same issue, and it is quite a show stopper.
wpoxon
commented
2 years ago I am not aware of a resolution to this issue, but I am no longer working on the project where I encountered this, so I am afraid I cannot provide any information about whether this may have been resolved by any changes in pytest-benchmark after my last update from Jan 29, 2021.
I am working on some code that I would like to contribute that implements a more compact set of X axis labels for histograms (a new format for --benchmark-name=FORMAT).
I am currently trying to get a clean run of tox against a baseline (unmodified) version of the current (3.2.3) pytest-benchmark code, and almost all the tests run cleanly with tox except on about 1/2 to 1/3 of the test environments I get errors because the coverage command fails because the .coverage file in the pytest-benchmark toplevel directory is zero length (so the coverage command fails because it cannot find the data it is expecting in the .coverage file).
At least 50% of the test environments in a tox run do not encounter this error, but a significant number of them do.
The error looks like this:
The summary at the end of the tox run looks like this:
I will look into the report and docs failures separately, for now I am only concerned amout the coverage failures. If there are specific log files that you would like me to attach to help diagnose the issue, please let me know.