Summary

the same command run locally vs. run in the Github CI produces vastly different coverage reports.

Expected vs actual result

running locally:

$ pytest -v --cov=./src/codecov_demo --cov=./tests --cov-config=pyproject.toml --cov-report=term-missing
=============================== test session starts ================================
platform win32 -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- C:\Users\ssiegmun\AppData\Local\hatch\env\virtual\codecov-demo\54RJULTp\codecov-demo\Scripts\python.EXE
cachedir: .pytest_cache
rootdir: C:\Users\ssiegmun\src\codecov-demo
plugins: cov-4.0.0
collected 4 items

tests/test_calculator.py::test_add PASSED                                     [ 25%] 
tests/test_calculator.py::test_subtract PASSED                                [ 50%] 
tests/test_calculator.py::test_multiply PASSED                                [ 75%] 
tests/test_calculator.py::test_divide PASSED                                  [100%]

---------- coverage: platform win32, python 3.10.4-final-0 -----------
Name                               Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------
src\codecov_demo\__about__.py          7      0      0      0   100%
src\codecov_demo\__init__.py           1      0      0      0   100%
src\codecov_demo\__main__.py           0      0      0      0   100%
src\codecov_demo\api\__init__.py       0      0      0      0   100%
src\codecov_demo\api\app.py           22     22      2      0     0%   3-37
src\codecov_demo\calculator.py        11      1      2      1    85%   15
tests\__init__.py                      0      0      0      0   100%
tests\test_calculator.py              25      0      0      0   100%
------------------------------------------------------------------------------
TOTAL                                 66     23      4      1    63%

but run in Gitlab CI:

  pytest -v --cov=./src/codecov_demo --cov=./tests --cov-config=pyproject.toml --cov-report=term-missing --cov-report=xml ./tests
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.0/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.0/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.0/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.0/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.0/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.0/x64/lib

============================= test session starts ==============================
platform linux -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0 -- /opt/hostedtoolcache/Python/3.11.0/x64/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/codecov-demo/codecov-demo
plugins: cov-4.0.0
collecting ... collected 4 items
tests/test_calculator.py::test_add PASSED                                [ 25%]
tests/test_calculator.py::test_subtract PASSED                           [ 50%]
tests/test_calculator.py::test_multiply PASSED                           [ 75%]
tests/test_calculator.py::test_divide PASSED                             [100%]
---------- coverage: platform linux, python 3.11.0-final-0 -----------
Name                               Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------
src/codecov_demo/__about__.py          7      7      0      0     0%   4-11
src/codecov_demo/__init__.py           1      1      0      0     0%   4
src/codecov_demo/__main__.py           0      0      0      0   100%
src/codecov_demo/api/__init__.py       0      0      0      0   100%
src/codecov_demo/api/app.py           22     22     10      0     0%   3-37
src/codecov_demo/calculator.py        11     11      2      0     0%   3-16
tests/__init__.py                      0      0      0      0   100%
tests/test_calculator.py              25      0      0      0   100%
------------------------------------------------------------------------------
TOTAL                                 66     41     12      0    32%
Coverage XML written to file coverage.xml
============================== 4 passed in 0.10s ===============================

Why is the CI result different from my local run?

Reproducer

Versions

pytest 7.2.0 Python 3.10.4

Config

pyproject.toml (partial):

[tool.hatch.envs.test]
dependencies = [
    "pytest",
    "pytest-cov",
]

[tool.hatch.envs.default.scripts]
cov = "pytest -vx --cov-report=term-missing --cov-config=pyproject.toml --cov=src/codecov_demo --cov=tests/"
no-cov = "cov --no-cov"

[[tool.hatch.envs.test.matrix]]
python = ["310"]

[tool.coverage.run]
branch = true
parallel = true
omit = [
  #"src/codecov_demo/__about__.py",
]

[tool.coverage.report]
exclude_lines = [
  "no cov",
  "if __name__ == .__main__.:",
  "if TYPE_CHECKING:",
]

Code

https://github.com/Nagidal/codecov-demo Github CI runs when some insignificant changes are pushed, for example to the "step3" branch. (I was following the Codecov-demo Github Tutorial)

What has been tried to solve the problem

Many different variations of the --cov parameters and the default working directory of the CI job

pytest-dev / pytest-cov

Different coverage locally than in Github CI #568