rowleya commented 4 years ago

Describe the bug When we run pytest using multiple processes with coverage on our tests, we get an error message that seems to be related to the integration of SQLite. The error we get is available from our integration test suite here: http://apollo.cs.man.ac.uk:8080/blue/organizations/jenkins/sPyNNaker8%20Integration%20Tests/detail/neuron_recorder_agdr/10/pipeline/102

Replicated a sample of the output here in case this disappears!

[gw7] linux -- Python 3.6.7 /usr/bin/python3
self = <SqliteDb @0x7f9efba0c1d0 debug=None filename='/var/lib/jenkins/workspace/ation_Tests_neuron_recorder_agdr/.coverage' nest=0 con=None>
sql = 'insert into meta (key, value) values (?, ?)'
parameters = ('has_arcs', '1')

    def execute(self, sql, parameters=()):
        """Same as :meth:`python:sqlite3.Connection.execute`."""
        if self.debug:
            tail = " with {!r}".format(parameters) if parameters else ""
            self.debug.write("Executing {!r}{}".format(sql, tail))
        try:
>           return self.con.execute(sql, parameters)
E           sqlite3.IntegrityError: UNIQUE constraint failed: meta.key

To Reproduce How can we reproduce the problem? Please be specific.

What version of Python are you using? 3.6
What version of coverage.py are you using? The output of coverage debug sys is helpful. 5.0
What versions of what packages do you have installed? The output of pip freeze is helpful. See above link for details.
What code are you running? Give us a specific commit of a specific repo that we can check out. Integration tests in Jenkins; see above for details.

What commands did you run?

py.test SpiNNUtils/unittests -rs -n auto --forked --show-progress --cov-branch --cov spynnaker8 --cov spynnaker --cov spinn_front_end_common --cov pacman --cov data_specification --cov spinnman --cov spinn_machine --cov spinn_storage_handlers --cov spalloc --cov spinn_utilities --junitxml junit/SpiNNUtils.xml --cov-report xml:coverage.xml --cov-append --timeout 1200

Expected behavior A clear and concise description of what you expected to happen. No error message; protection against writing from mulitple processes through checking for key in database before attempted insert.

Additional context Add any other context about the problem here.

kstaniek commented 4 years ago

Facing the same issue. Is there any workaround, pls?

nedbat commented 4 years ago

The simplest way to avoid this issue for now is to pin coverage.py to 4.5.4. Another thing that would help me find a solution for you is to provide me with specific, complete instructions that would let me reproduce the problem.

nedbat commented 4 years ago

@rowleya Can you give me specific instructions I can follow to reproduce the problem? What repo, what commit, what commands to run?

AlexandreYang commented 4 years ago

I couldn't reproduce reliably the issue we have on this CI Pipeline https://dev.azure.com/datadoghq/integrations-core/_build/results?buildId=4968&view=logs&jobId=6856723d-645b-5a7b-98a0-8af874afe33f&j=82f604ee-2edd-5639-6463-dbdc7756af1f&t=1d9be0b0-41b5-5871-19d1-c01bf70ecbd1

But here are few things that might help:

The bug happen when using pytest with both --cov-append --cov=X options. Without --cov-append, it works fine.
When the error sqlite3.IntegrityError: UNIQUE constraint failed: meta.key is raised (while inserting a new entry with has_arcs key in meta table), there is already an entry in the meta table with ('has_arcs', '0')

rowleya commented 4 years ago

The specific instructions that seem to fail for me are:

python3.6 -m virtualenv test
cd test
source bin/activate
git clone https://github.com/SpiNNakerManchester/SpiNNUtils.git
cd SpiNNUtils && python setup.py develop && cd ..
pip install python-coveralls "coverage>=4.4"
pip install pytest-instafail pytest-xdist pytest-cov pytest-progress pytest-timeout pytest-forked
py.test SpiNNUtils/unittests -rs -n auto --forked --show-progress --cov-branch --cov spynnaker8 --cov spynnaker --cov spinn_front_end_common --cov pacman --cov data_specification --cov spinnman --cov spinn_machine --cov spinn_storage_handlers --cov spalloc --cov spinn_utilities --junitxml junit/SpiNNUtils.xml --cov-report xml:coverage.xml --cov-append --timeout 1200

rowleya commented 4 years ago

It makes sense that --cov-append is the issue; I am trying to achieve something specific with this, which is to run lots of tests from different repositories, including integration tests, and get the overall coverage, so this is essential.

A quick test shows that without the -n auto --forked but with the other arguments (including --cov-append) this also works. It fails some of the tests due to the above not installing all the requirements, but the coverage module doesn't fail; the extra dependencies are:

pip install numpy testfixtures httpretty

Urth commented 4 years ago

I found a reproducible configuration which was plaguing one of our projects. The coverage database/internals will enter a broken state when sub-process coverage is used. A minimal example can be found at https://github.com/Urth/coveragepy_883

sco1 commented 4 years ago

I believe I might be seeing a similar issue in one of my latest builds, though in our case it seems specific to having a test that invokes subprocess.

Here is the specific commit that triggers the CI failure (3.6 log, azure build link, tox config). While this fails in CI for 3.6, 3.7, and 3.8, on my MBP it only fails for 3.6. Removing the test_flake8_actually_runs_checker.py test (the one with subprocess) eliminates the failure.

I will work on capturing more details, but might not be able to get them up until tomorrow.

nedbat commented 4 years ago

This works to reproduce the problem:

git clone https://github.com/python-discord/flake8-annotations
cd flake8-annotations
git checkout 1dfaab8100a686e2171b0e27d0c21cc195be4db6
mktmpenv -p python3.6 -n
pip install tox
rm .coverage; tox -e py36 -- testing/test_flake8_actually_runs_checker.py

nedbat commented 4 years ago

BTW,

While this fails in CI for 3.6, 3.7, and 3.8, on my MBP it only fails for 3.6.

The test fails if there is no .coverage file, which is why CI fails always, but only the first tox env fails on your laptop. With "rm .coverage", it fails on all versions of Python locally.

nedbat commented 4 years ago

One solution is to add this to a .coveragerc file:

[run]
parallel = True

Removing --cov-append also prevents the error. I'm not sure which of these gives you the proper measurements, if either. I don't know if there's a better solution that can be built in to coverage.py or to pytest-cov.

sco1 commented 4 years ago

Thanks Ned, away from a dev machine until this evening to check this out, but I don't think I'm following why the missing coverage file is ultimate source of the problem.

I can't speak for the OP, but for my project if the malfunctioning test is removed then the test suite runs fine, both locally and in CI. For example, the latest release commit (CI) runs without issue.

edit: I've added the suggested .coveragerc file and it seems to resolve the issue, yay!

JulienPalard commented 4 years ago

I can reproduce this issue with tox, running like:

while sleep 1; do tox -p all -e py36,py37 -vv; done

with this testenv:

[testenv]
deps = -r dev-requirements.txt
commands = pytest -v --capture=fd --cov-report term-missing --cov={envsitepackagesdir}/mpi tests

Although it happen only once every 10 run, more or less.

Adding

parallel = True

in a tox.ini file does not resolve it. adding:

[run]
parallel = True

to .coveragerc looks to fix it at first, but running my while long enough gets me the bug back from time to time (had to wait like 30 runs before seeing it once).

here is a full tox log

``` cleanup /home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/14/mypi-0.6.3.tar.gz using tox.ini: /home/mdk/clones/JulienPalard/mypi/tox.ini (pid 1550551) removing /home/mdk/clones/JulienPalard/mypi/.tox/log /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 (/home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6) is {'executable': '/home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6', 'name': 'python', 'version_info': [3, 6, 9, 'final', 0], 'version': '3.6.9 (default, Dec 12 2019, 09:56:25) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} .package uses /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 py36 uses /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 python3.7 (/home/mdk/.local/bin/python3.7) is {'executable': '/home/mdk/.local/bin/python3.7', 'name': 'python', 'version_info': [3, 7, 5, 'final', 0], 'version': '3.7.5 (default, Dec 12 2019, 09:56:15) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} py37 uses /home/mdk/.local/bin/python3.7 python3.8 (/home/mdk/.local/bin/python3.8) is {'executable': '/home/mdk/.local/bin/python3.8', 'name': 'python', 'version_info': [3, 8, 0, 'final', 0], 'version': '3.8.0 (default, Dec 12 2019, 09:56:28) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} py38 uses /home/mdk/.local/bin/python3.8 using tox-3.14.3 from /home/mdk/clones/JulienPalard/mypi/.venv36/lib/python3.6/site-packages/tox/__init__.py (pid 1550551) .package start: getenv /home/mdk/clones/JulienPalard/mypi/.tox/.package .package reusing: /home/mdk/clones/JulienPalard/mypi/.tox/.package .package finish: getenv /home/mdk/clones/JulienPalard/mypi/.tox/.package after 0.05 seconds .package start: finishvenv .package finish: finishvenv after 0.03 seconds .package start: get-build-requires /home/mdk/clones/JulienPalard/mypi/.tox/.package setting PATH=/home/mdk/clones/JulienPalard/mypi/.tox/.package/bin:/home/mdk/clones/JulienPalard/mypi/.venv36/bin:/home/mdk/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games [1550569] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.tox/.package/bin/python .venv36/lib/python3.6/site-packages/tox/helper/build_requires.py setuptools.build_meta '' >.tox/.package/log/.package-247.log .package finish: get-build-requires /home/mdk/clones/JulienPalard/mypi/.tox/.package after 0.15 seconds .package start: perform-isolated-build /home/mdk/clones/JulienPalard/mypi/.tox/.package setting PATH=/home/mdk/clones/JulienPalard/mypi/.tox/.package/bin:/home/mdk/clones/JulienPalard/mypi/.venv36/bin:/home/mdk/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games [1550572] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.tox/.package/bin/python .venv36/lib/python3.6/site-packages/tox/helper/build_isolated.py .tox/dist setuptools.build_meta '' >.tox/.package/log/.package-248.log running sdist running egg_info writing mypi.egg-info/PKG-INFO writing dependency_links to mypi.egg-info/dependency_links.txt writing entry points to mypi.egg-info/entry_points.txt writing requirements to mypi.egg-info/requires.txt writing top-level names to mypi.egg-info/top_level.txt reading manifest file 'mypi.egg-info/SOURCES.txt' writing manifest file 'mypi.egg-info/SOURCES.txt' running check creating mypi-0.6.3 creating mypi-0.6.3/mypi creating mypi-0.6.3/mypi.egg-info copying files to mypi-0.6.3... copying README.md -> mypi-0.6.3 copying pyproject.toml -> mypi-0.6.3 copying setup.cfg -> mypi-0.6.3 copying setup.py -> mypi-0.6.3 copying mypi/__init__.py -> mypi-0.6.3/mypi copying mypi/hal.py -> mypi-0.6.3/mypi copying mypi/middlewares.py -> mypi-0.6.3/mypi copying mypi/models.py -> mypi-0.6.3/mypi copying mypi/product_api.py -> mypi-0.6.3/mypi copying mypi/search.py -> mypi-0.6.3/mypi copying mypi/views.py -> mypi-0.6.3/mypi copying mypi.egg-info/PKG-INFO -> mypi-0.6.3/mypi.egg-info copying mypi.egg-info/SOURCES.txt -> mypi-0.6.3/mypi.egg-info copying mypi.egg-info/dependency_links.txt -> mypi-0.6.3/mypi.egg-info copying mypi.egg-info/entry_points.txt -> mypi-0.6.3/mypi.egg-info copying mypi.egg-info/requires.txt -> mypi-0.6.3/mypi.egg-info copying mypi.egg-info/top_level.txt -> mypi-0.6.3/mypi.egg-info Writing mypi-0.6.3/setup.cfg Creating tar archive removing 'mypi-0.6.3' (and everything under it) mypi-0.6.3.tar.gz .package finish: perform-isolated-build /home/mdk/clones/JulienPalard/mypi/.tox/.package after 0.16 seconds copying new sdistfile to '/home/mdk/.tox/distshare/mypi-0.6.3.tar.gz' package .tmp/package/14/mypi-0.6.3.tar.gz links to dist/mypi-0.6.3.tar.gz (/home/mdk/clones/JulienPalard/mypi/.tox) ⠋ [0] py36 start: parallel py36 py37 start: parallel py37 [1550578] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 .venv36/lib/python3.6/site-packages/tox/__main__.py -p all -e py36,py37 -vv --installpkg .tox/.tmp/package/14/mypi-0.6.3.tar.gz >.tox/py36/log/py36-308.log [1550579] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 .venv36/lib/python3.6/site-packages/tox/__main__.py -p all -e py36,py37 -vv --installpkg .tox/.tmp/package/14/mypi-0.6.3.tar.gz >.tox/py37/log/py37-142.log ⠴ [2] py36 | py37ERROR: invocation failed (exit code 1), logfile: /home/mdk/clones/JulienPalard/mypi/.tox/py36/log/py36-308.log ========================================================================== log start ========================================================================== using tox.ini: /home/mdk/clones/JulienPalard/mypi/tox.ini (pid 1550578) /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 (/home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6) is {'executable': '/home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6', 'name': 'python', 'version_info': [3, 6, 9, 'final', 0], 'version': '3.6.9 (default, Dec 12 2019, 09:56:25) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} .package uses /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 py36 uses /home/mdk/clones/JulienPalard/mypi/.venv36/bin/python3.6 python3.7 (/home/mdk/.local/bin/python3.7) is {'executable': '/home/mdk/.local/bin/python3.7', 'name': 'python', 'version_info': [3, 7, 5, 'final', 0], 'version': '3.7.5 (default, Dec 12 2019, 09:56:15) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} py37 uses /home/mdk/.local/bin/python3.7 python3.8 (/home/mdk/.local/bin/python3.8) is {'executable': '/home/mdk/.local/bin/python3.8', 'name': 'python', 'version_info': [3, 8, 0, 'final', 0], 'version': '3.8.0 (default, Dec 12 2019, 09:56:28) \n[GCC 9.2.1 20191130]', 'is_64': True, 'sysplatform': 'linux'} py38 uses /home/mdk/.local/bin/python3.8 using tox-3.14.3 from /home/mdk/clones/JulienPalard/mypi/.venv36/lib/python3.6/site-packages/tox/__init__.py (pid 1550578) using package '/home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/14/mypi-0.6.3.tar.gz', skipping 'sdist' activity package 16/mypi-0.6.3.tar.gz links to 14/mypi-0.6.3.tar.gz (/home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package) py36 start: getenv /home/mdk/clones/JulienPalard/mypi/.tox/py36 py36 reusing: /home/mdk/clones/JulienPalard/mypi/.tox/py36 py36 finish: getenv /home/mdk/clones/JulienPalard/mypi/.tox/py36 after 0.07 seconds py36 start: installpkg /home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/16/mypi-0.6.3.tar.gz py36 inst-nodeps: /home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/16/mypi-0.6.3.tar.gz setting PATH=/home/mdk/clones/JulienPalard/mypi/.tox/py36/bin:/home/mdk/clones/JulienPalard/mypi/.venv36/bin:/home/mdk/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games [1550615] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.tox/py36/bin/python -m pip install --no-deps -U .tox/.tmp/package/16/mypi-0.6.3.tar.gz Processing ./.tox/.tmp/package/16/mypi-0.6.3.tar.gz Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'done' Building wheels for collected packages: mypi Building wheel for mypi (PEP 517): started Building wheel for mypi (PEP 517): finished with status 'done' Created wheel for mypi: filename=mypi-0.6.3-cp36-none-any.whl size=10179 sha256=379aac002d64d204e8b219f4356bcc68558ee8ebca969de2f88802f9c381acc7 Stored in directory: /home/mdk/.cache/pip/wheels/09/8b/4e/f7e5a1721335e2eb7f30d5353ab65d2166dc867ce5269accdf Successfully built mypi Installing collected packages: mypi Found existing installation: mypi 0.6.3 Not uninstalling mypi at /home/mdk/clones/JulienPalard/mypi, outside environment /home/mdk/clones/JulienPalard/mypi/.tox/py36 Can't uninstall 'mypi'. No files were found to uninstall. Successfully installed mypi-0.6.3 py36 finish: installpkg /home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/16/mypi-0.6.3.tar.gz after 1.99 seconds py36 start: envreport setting PATH=/home/mdk/clones/JulienPalard/mypi/.tox/py36/bin:/home/mdk/clones/JulienPalard/mypi/.venv36/bin:/home/mdk/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games [1550664] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.tox/py36/bin/python -m pip freeze >.tox/py36/log/py36-309.log py36 finish: envreport after 0.27 seconds py36 installed: aiohttp==3.6.2,aiohttp-cors==0.7.0,appdirs==1.4.3,astroid==2.3.3,async-timeout==3.0.1,attrs==19.3.0,bandit==1.6.2,black==19.10b0,bumpversion==0.5.3,chardet==3.0.4,Click==7.0,coverage==5.0.3,defusedxml==0.6.0,entrypoints==0.3,filelock==3.0.12,flake8==3.7.9,freezegun==0.3.13,gitdb2==2.0.6,GitPython==3.0.5,idna==2.8,idna-ssl==1.1.0,importlib-metadata==1.4.0,isort==4.3.21,lazy-object-proxy==1.4.3,mccabe==0.6.1,more-itertools==8.1.0,multidict==4.7.4,mypy==0.761,mypy-extensions==0.4.3,packaging==20.0,pathspec==0.7.0,pbr==5.4.4,pluggy==0.13.1,py==1.8.1,pycodestyle==2.5.0,pyflakes==2.1.1,pylint==2.4.4,pyparsing==2.4.6,pytest==5.3.3,pytest-aiohttp==0.3.0,pytest-cov==2.8.1,python-dateutil==2.8.1,python-slugify==4.0.0,PyYAML==5.3,regex==2020.1.8,mypi==0.6.3,six==1.14.0,smmap2==2.0.5,stevedore==1.31.0,text-unidecode==1.3,toml==0.10.0,tox==3.14.3,typed-ast==1.4.1,typing-extensions==3.7.4.1,virtualenv==16.7.9,wcwidth==0.1.8,wrapt==1.11.2,yarl==1.4.2,zipp==1.0.0 removing /home/mdk/clones/JulienPalard/mypi/.tox/py36/tmp py36 start: run-test-pre py36 run-test-pre: PYTHONHASHSEED='758267525' py36 finish: run-test-pre after 0.00 seconds py36 start: run-test py36 run-test: commands[0] | pytest -v --capture=fd --cov-report term-missing --cov=/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/mypi tests setting PATH=/home/mdk/clones/JulienPalard/mypi/.tox/py36/bin:/home/mdk/clones/JulienPalard/mypi/.venv36/bin:/home/mdk/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games [1550666] /home/mdk/clones/JulienPalard/mypi$ /home/mdk/clones/JulienPalard/mypi/.tox/py36/bin/pytest -v --capture=fd --cov-report term-missing --cov=/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/mypi tests ============================= test session starts ============================== platform linux -- Python 3.6.9, pytest-5.3.3, py-1.8.1, pluggy-0.13.1 -- /home/mdk/clones/JulienPalard/mypi/.tox/py36/bin/python cachedir: .tox/py36/.pytest_cache rootdir: /home/mdk/clones/JulienPalard/mypi plugins: aiohttp-0.3.0, cov-2.8.1 collecting ... collected 24 items tests/test_base.py::test_slash[pyloop] PASSED [ 4%] tests/test_base.py::test_without_xml[pyloop] PASSED [ 8%] tests/test_base.py::test_health_red_orange_green[pyloop] PASSED [ 12%] tests/test_base.py::test_empty_xml[pyloop] PASSED [ 16%] tests/test_base.py::test_brand[pyloop] PASSED [ 20%] tests/test_base.py::test_category[pyloop] PASSED [ 25%] tests/test_base.py::test_product[pyloop] PASSED [ 29%] tests/test_base.py::test_initial_price[pyloop] PASSED [ 33%] tests/test_base.py::test_product_not_found[pyloop] PASSED [ 37%] tests/test_base.py::test_multi_product[pyloop] PASSED [ 41%] tests/test_base.py::test_brands[pyloop] PASSED [ 45%] tests/test_base.py::test_categories[pyloop] PASSED [ 50%] tests/test_base.py::test_products_do_not_list_all[pyloop] PASSED [ 54%] tests/test_base.py::test_products_search[pyloop] PASSED [ 58%] tests/test_base.py::test_products_search_by_brand[pyloop] PASSED [ 62%] tests/test_base.py::test_products_search_by_url[pyloop] PASSED [ 66%] tests/test_base.py::test_products_search_exact[pyloop] PASSED [ 70%] tests/test_base.py::test_products_search_by_sku[pyloop] PASSED [ 75%] tests/test_base.py::test_parse_missing_args PASSED [ 79%] tests/test_base.py::test_parse_args PASSED [ 83%] tests/test_base.py::test_main PASSED [ 87%] tests/test_caracteristic_detection.py::test_ml PASSED [ 91%] tests/test_i18n.py::test_i18n_products_search[pyloop] PASSED [ 95%] tests/test_i18n.py::test_i18n_products_search_traverse_language[pyloop] PASSED [100%] INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/sqldata.py", line 1046, in execute INTERNALERROR> return self.con.execute(sql, parameters) INTERNALERROR> sqlite3.IntegrityError: UNIQUE constraint failed: meta.key INTERNALERROR> INTERNALERROR> During handling of the above exception, another exception occurred: INTERNALERROR> INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/_pytest/main.py", line 196, in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/_pytest/main.py", line 246, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__ INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pluggy/manager.py", line 87, in INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall INTERNALERROR> gen.send(outcome) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pytest_cov/plugin.py", line 254, in pytest_runtestloop INTERNALERROR> self.cov_controller.finish() INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/pytest_cov/engine.py", line 197, in finish INTERNALERROR> self.cov.stop() INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/control.py", line 680, in combine INTERNALERROR> combine_parallel_data(self._data, aliases=aliases, data_paths=data_paths, strict=strict) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/data.py", line 117, in combine_parallel_data INTERNALERROR> data.update(new_data, aliases=aliases) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/sqldata.py", line 689, in update INTERNALERROR> self._choose_lines_or_arcs(arcs=True) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/sqldata.py", line 497, in _choose_lines_or_arcs INTERNALERROR> ('has_arcs', str(int(arcs))) INTERNALERROR> File "/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/coverage/sqldata.py", line 1063, in execute INTERNALERROR> raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, msg)) INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/home/mdk/clones/JulienPalard/mypi/.coverage': UNIQUE constraint failed: meta.key ============================== 24 passed in 0.60s ============================== ERROR: InvocationError for command /home/mdk/clones/JulienPalard/mypi/.tox/py36/bin/pytest -v --capture=fd --cov-report term-missing --cov=/home/mdk/clones/JulienPalard/mypi/.tox/py36/lib/python3.6/site-packages/mypi tests (exited with code 3) py36 finish: run-test after 0.99 seconds py36 start: run-test-post py36 finish: run-test-post after 0.00 seconds cleanup /home/mdk/clones/JulienPalard/mypi/.tox/.tmp/package/16/mypi-0.6.3.tar.gz =========================================================================== log end =========================================================================== ✖ FAIL py36 in 3.594 seconds ⠦ [1] py37py37 finish: parallel py37 after 3.67 seconds ✔ OK py37 in 3.674 seconds ___________________________________________________________________________ summary ___________________________________________________________________________ ERROR: py36: parallel child exit code 1 py37: commands succeeded ```

Looks like both coverage processes are trying to work on the same sqlite file.

rowleya commented 4 years ago

For me it seems that the suggestion fixes the issue.

[run]
parallel = True

Thanks!

xavfernandez commented 4 years ago

Is there any drawback to using:

[run]
parallel = True

?

And if there isn't, why is it not the default ? (honest question)

nedbat commented 4 years ago

The drawback to parallel = True is that you need an explicit coverage combine step before you can use coverage report or coverage html (or any other reporting).

MartinThoma commented 4 years ago

Just to highlight: Adding the following section to the setup.cfg fixes the issue:

[coverage:run]
parallel=true

edit: Strange ... I think this worked a while ago. Now it doesn't anymore.

KingDarBoja commented 4 years ago

Having the same issue since the update to 5.0.4. On 5.0.3 is working fine on GitHub Actions 😢

Does the parallel option is required if I am trying to do something like below?

pytest --cov-report= --cov=test_file1.py -vv &
pytest --cov-report= --cov=test_file2.py --cov-append -vv &
pytest --cov-report= --cov=test_file3.py --cov-append -vv &
coverage report -m

JulienPalard commented 4 years ago

TL;DR

The issue happen when running multiple coverage combine in parallel from multiple distincts coverage runs, as there is no way for coverage to tell which .coverage.* files are from which set of coverage run. It happen typically when using pytest-cov in parallel, as pytest-cov always uses a combine step, even when a single .coverage file is needed.

The way to tell coverage which set of .coverage.* file are from which set of runs (with or without tox), is to specify the .coverage filename. For tox, one can use:

setenv =
  COVERAGE_FILE=.coverage.{envname}

I think this issue can be closed, but pytest-cov may have to document this better.

Long version

I'm trying to reproduce it today, I'm able to reproduce it with either tox -p all and:

$ /tmp/3.6/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/mediaserver/ &
/tmp/3.7/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/mediaserver/ &
/tmp/3.8/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/mediaserver/

It looks like it's always an issue near .coverage, as I got those errors:

coverage.misc.CoverageException: Couldn't use data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.40820.348297': no such table: file

or

INTERNALERROR> coverage.misc.CoverageException: Data file '/home/mdk/clones/coverage-reproducer/.coverage' doesn't seem to be a coverage data file: Couldn't use data file '/home/mdk/clones/coverage-reproducer/.coverage': no such table: coverage_schema

Interestingly I got a no such table: file in .coverage.seraph.41964.820593 while not using --parallel-mode. Happen that pytest-cov is adding it.

stacktrace

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/coverage/sqldata.py", line 1048, in execute
INTERNALERROR>     return self.con.execute(sql, parameters)
INTERNALERROR> sqlite3.OperationalError: no such table: file
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/_pytest/main.py", line 191, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/_pytest/main.py", line 247, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pluggy/manager.py", line 87, in 
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pytest_cov/engine.py", line 46, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/pytest_cov/engine.py", line 231, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/coverage/control.py", line 687, in combine
INTERNALERROR>     combine_parallel_data(self._data, aliases=aliases, data_paths=data_paths, strict=strict)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/coverage/data.py", line 117, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/coverage/sqldata.py", line 575, in update
INTERNALERROR>     cur = conn.execute('select path from file')
INTERNALERROR>   File "/tmp/3.6/lib/python3.6/site-packages/coverage/sqldata.py", line 1065, in execute
INTERNALERROR>     raise CoverageException("Couldn't use data file {!r}: {}".format(self.filename, msg))
INTERNALERROR> coverage.misc.CoverageException: Couldn't use data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.41964.820593': no such table: file

I see that pytest-cov is instanciating two Coverage, which may lead to the two sqlite file being overwritten.

I'm trying with 3 pytest-cov in parallel, with export COVERAGE_DEBUG=dataio, what I'm getting is surprising:

First process (everything OK)

Erasing data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46134.640334'
Creating data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46134.640334'
Combining data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46134.640334'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46134.640334'
Erasing data file '/home/mdk/clones/coverage-reproducer/.coverage'
Creating data file '/home/mdk/clones/coverage-reproducer/.coverage'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Deleting combined data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46134.640334'

2nd process (get mixed up with 3st one)

Erasing data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'
Creating data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Combining data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Deleting combined data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Combining data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'

3rd process (get mixed up with 2st one)

Erasing data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Creating data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Combining data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Deleting combined data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46132.615372'
Combining data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'
Opening data file '/home/mdk/clones/coverage-reproducer/.coverage'
Deleting combined data file '/home/mdk/clones/coverage-reproducer/.coverage.seraph.46133.288847'

my test script

export COVERAGE_DEBUG=dataio
/tmp/3.6/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/ > 1.log 2>1.err &
/tmp/3.7/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/ > 2.log 2>2.err &
/tmp/3.8/bin/pytest --cov-report term-missing --cov-fail-under=100 --cov src/ > 3.log 2>3.err
wait

Looks like the problem tighten up around pytest-cov using coverage combine, which can't really work when multiple process are spawning files in parallel.

Reproducer https://mdk.fr/x/pytest-cov-reproducer.tar.bz2, which I run using while tox -p all; do :; done and wait (it can take from a few seconds to ... I don't know sorry).

JulienPalard commented 3 years ago

Following up on this one, I successfully use coverage in parallel since enough time to tell it's stable, but without pytest-cov which uses an intermediate combine step (see previous message).

An example using tox to run pytest and coverage for multiple Python versions in parallel and then combining the result can be found here:

https://github.com/JulienPalard/oeis/blob/14bea829d556bd617426c8a85235ba5da1b9f157/tox.ini#L33

I think this issue can be closed.

nedbat commented 2 years ago

I believe this is now fixed in commit b41be3f9

nedbat commented 2 years ago

This is now released as part of coverage 6.3.

bulat15g commented 2 years ago

coverage==6.3.1 no errors. Thank you

nedbat / coveragepy

Coverage in parallel failing #883

TL;DR

Long version