Running pytest-cov in parallel

[JulienPalard]

Summary

People are using tox -p auto or tox -p all more and more since it exists, (and some are simply using & in shell scripts).

But it's failing with pytest-cov (https://github.com/nedbat/coveragepy/issues/883, https://github.com/pytest-dev/pytest-cov/issues/356, https://github.com/pytest-dev/pytest-cov/issues/237, https://github.com/pytest-dev/pytest-cov/issues/217).

This is because pytest-cov uses a coverage combine step which tries to combine all .coverage.* files, mixing files from all of the parallels runs. As some are incomplete, this often yield to sqlite errors, but it also sometime just mix the data in strange ways.

A clean fix is to specify a specific coverage file name for each run, so the combine step will search for files with this specific name, avoiding mixing the files.

This can easily be done, for example, in .tox.ini by using:

setenv =
  COVERAGE_FILE=.coverage.{envname}

It make coverage combine search for .coverage.py37.* for example.

I see two strategies:

Either pytest-cov picks a unique coverage file name per run or pytest-cov documents that when used in parallel one should specify a coverage file name to disambiguate the runs.

Do you have a preference?

[JulienPalard]

As I read a few issues, it looks like many people are relying on a .coverage being generated, so they can typically upload them somewhere, which leaves an open question to me:

Imagine one have a tox.ini like:

[tox]
envlist = py37, py38

[testenv]
deps = -rrequirements.txt
commands = pytest --cov-report term-missing --cov {envsitepackagesdir}/foo/

and some Python code like:

if sys.version_info >= (3, 8):
    ...
else:
    ...

I don't think we want two distinct coverage files, one whining about the first branch not being tested and the other whining about the other branch not being tested.

I think we need a single .coverage file telling that all branches are tested, a "final combination", which, I think, can't be obtained right now as far as I understand.

Using my proposed

  COVERAGE_FILE=.coverage.{envname}

gives 3 output files, .coverage.py37, .coverage.py38, .coverage.py39, which can be combined manually with coverage combine, which correctly report 100% coverage.

I'm trying with this "reproducer" https://mdk.fr/x/pytest-cov-reproducer.tar.bz2

Haven't found a way to tell tox "run this once all parallel jobs are done", which would anyay be tox side, not pytest-cov side.

So I think we can only document all of this and let user choose what they want according to what they need.

[cadedaniel]

Hi, I am a pytest-cov user.

Haven't found a way to tell tox "run this once all parallel jobs are done", which would anyay be tox side, not pytest-cov side.

So I think we can only document all of this and let user choose what they want according to what they need.

This would solve my problem exactly -- I am not using tox, and can afford making a call after-the-fact to combine the results. If I can disable the automatic combining via flag/config, it will make pytest-cov more composable with other paradigms (like MPI).

[ionelmc]

So in theory there could be that option, that you can use and get the old broken behavior. But the old broken behavior still read other coverage files (for subprocess support). So then you'd ask how about an option to disable that too right? But then you'd basically make pytest-cov do almost nothing, and you might as well just use coverage run -mpytest or similar.

I would argue that running tests in parallel with coverage on is a fringe usecase - make up your mind and chose either speed or coverage.

[JulienPalard]

I solved my issue of running tox --parallel=all, having coverage done on all parallel runs, and merge them at the end, see: https://github.com/JulienPalard/oeis/blob/master/tox.ini

[adam-grant-hendry]

The fact pytest-cov does not support the parallel option is a deal-breaker for me. I originally wanted to use the plugin out of convenience, but since it creates a headache of a discrepancy between local vs. CI results, it's much easier for me to switch back to running

coverage run -m pytest

to get coverage results instead of

pytest

and having the pytest-cov plugin run coverage.

I'm uninstalling pytest-cov. Even the coverage.py docs state:

Many people choose to use the pytest-cov plugin, but for most purposes, it is unnecessary.

[zmievsa]

I just switched from pytest-cov back to solo coverage for the reason above.