Concurrency bugs in codecov (coverage.py) workflow

mkoeppe commented 10 months ago

Combined data file .coverage/.coverage.Helios.local.70384.568987
Skipping duplicate data .coverage/.coverage.Helios.local.71097.566208
Skipping duplicate data .coverage/.coverage.Helios.local.71074.300096

We also see these messages in large quantities the Build & Test CI runs, in the section "Prepare coverage results". For example https://github.com/sagemath/sage/actions/runs/6623852735/job/17991733088#step:19:20

/Users/kwankyu/GitHub/sage-dev/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/coverage/data.py:171: CoverageWarning: Data file '/Users/kwankyu/GitHub/sage-dev/src/.coverage/.coverage.Helios.local.39475.460681' doesn't seem to be a coverage data file: 
  data._warn(str(exc))
Couldn't combine data file .coverage/.coverage.Helios.local.39475.460681: Data file '/Users/kwankyu/GitHub/sage-dev/src/.coverage/.coverage.Helios.local.39475.460681' doesn't seem to be a coverage data file:

This one also shows up in the CI, for example https://github.com/sagemath/sage/actions/runs/6606391152/job/17942563622#step:19:1162, but with much less frequency.

This looks to me like a concurrency bug. It may, in fact, help explain the nondeterministic coverage results that we observed even after fixing the random seed. See #35522.

Originally posted by @mkoeppe in https://github.com/sagemath/sage/issues/36505#issuecomment-1777824058

mkoeppe commented 10 months ago

@tobiasdiez You already reacted in https://github.com/sagemath/sage/pull/36505#issuecomment-1779293008, but perhaps you could try to explain what you intended with the settings in https://github.com/sagemath/sage/blob/develop/src/tox.ini#L262 regarding multiprocessing.

tobiasdiez commented 10 months ago

This adds support for multiprocessing. Otherwise codecov doesn't collect any coverage in sub-threads.

sagemath / sage

Concurrency bugs in codecov (coverage.py) workflow #36539