Closed inkychris closed 2 years ago
Have you tried using "--parallel=True", and then "coverage combine" with a "[paths]" setting to rewrite the paths as you need?
I've been manually renaming the coverage files in their respective CI jobs which I think covers the --parallel=True
option. I'm not sure that the [paths]
section addresses this problem though, as the path relative to the root source directory is the bit that need rewriting, unless that is doable with this section of the config?
I've updated the description to include an example report generated on Windows from 4 test runs (2 Linux, 2 Windows). This doesn't work at all trying to report on Linux because it can't find any files with backslashes in, while Windows happily deals with the forward slashes used in the Linux results.
I can confirm that the following paths
-based configuration works well for combining coverage data across platforms:
[tool.coverage.paths]
source = ["src", "*/site-packages"]
tests = ["tests", "*/tests"]
[tool.coverage.run]
source = ["yourpackage", "tests"]
This assumes a project where the package under test is located under src/yourpackage
, and the test suite under tests
. It also assumes that the package is installed into the virtual environment where the tests run. Editable installs are not used.
As for the original issue, I can confirm that as well. Since relative_files
does not normalize path separators, combining coverage data across platforms using relative_files
instead of paths
results in errors like this:
No source for code: '/Users/cjolowicz/.../yourpackage/tests\__init__.py'.
Note the mixed forward and backward slashes in the path. This was run on macOS using both the latest release and master of Coverage.py.
I have attached the coverage data from the GitHub Action runners. Use these commands to reproduce:
git clone https://github.com/cjolowicz/cookiecutter-hypermodern-python-instance
cd cookiecutter-hypermodern-python-instance
git switch --detach 11b803bd73c6f702aeaa477f2eba78f4b101950f
unzip coverage-data.zip
coverage combine
coverage report
Below is a full traceback for the No source
error.
FWIW the paths in the combined coverage look like this:
❯ sqlite3 .coverage 'select * from file'
1|tests/__init__.py
2|tests/test_main.py
3|src/cookiecutter_hypermodern_python_instance/__init__.py
4|src/cookiecutter_hypermodern_python_instance/__main__.py
5|tests\__init__.py
6|tests\test_main.py
I'm running into a similar problem. I have coverage data generated in a Linux container, but want to generate the report on the Windows host. Since the Linux container wrote /
separators, the Windows host cannot find the source files when generating the report.
In case it helps anyone else: I've been able to work around this by re-writing the Windows coverage file prior to coverage combine
. As long as you use relative_files = True
, it's a 1 liner in sqlite3:
sqlite3 .coverage.* "update file set path = replace(path, '\\', '/')"
If you aren't using relative_files = True
, I imagine a very similar command would work; however, you'd also need to modify your [paths]
definition to account for the fact that the Windows path that is being merged will be using C:/path/to/src
rather than C:\path\to\src
.
Thanks for bring this to back to my attention, and sorry for letting it languish.
Definitely one thing is needed to fix this: if relative_files=True
, then an entry in [paths]
like foo
should remain foo
instead of being made absolute (/home/runner/project/foo
or whatever). That's this:
--- a/coverage/files.py
+++ b/coverage/files.py
@@ -363,8 +363,9 @@ def add(self, pattern, result):
# The pattern is meant to match a filepath. Let's make it absolute
# unless it already is, or is meant to match any prefix.
- if not pattern.startswith('*') and not isabs_anywhere(pattern + pattern_sep):
- pattern = abs_file(pattern)
+ if not self.relative:
+ if not pattern.startswith('*') and not isabs_anywhere(pattern + pattern_sep):
+ pattern = abs_file(pattern)
if not pattern.endswith(pattern_sep):
pattern += pattern_sep
Then if you had an entry like */tests
(as @cjolowicz does), you could change it to tests
instead, and it would work properly. But maybe it should also be that if using relative_files, then a pattern entry like */tests
should match tests/foo/bar.etc
. Today it doesn't because there's no slash before tests. In fact, maybe it should always match that? Doesn't */tests
mean, a "tests" directory anywhere? Opinions?
Actually, we don't even need the change in my diff if */foo
will match foo/bar/baz.etc
.
@nedbat That may be a bug, but I don't think it's related to the problem being described here (or, if it is, I'm not seeing the connection).
The issue I addressed with that SQLite hack isn't anything to do with the prefix; relative_paths
is a bit of a red herring. The issue is Windows vs macOS/Unix path separators.
If you have a coverage file generated on macOS/Unix, the coverage data will refer to files as /path/to/tests/foo/bar.py
; if you're on Windows, the coverage data will refer to C:\path\to\tests\foo\bar.py
; even if you use prefixes to make /path/to/tests
and C:\path\to\tests
equivalent, the coverage data sees foo/bar.py
and foo\bar.py
as two separate locations. When you merge macOS + Windows coverage data and then generate a HTML report, one of those two locations won't exist - because /path/to/tests/foo\bar.py
won't exist on macOS/Unix, and C:\path\to\tests\foo/bar.py
won't exist on Windows - and that's the error that is reported:
(venv3.10) rkm@stirtoni briefcase % coverage combine
Combined data file .coverage.RUSSELLKEITA152.4780.453503
Combined data file .coverage.stirtoni.local.27867.230158
(venv3.10) rkm@stirtoni briefcase % coverage report
No source for code: '/Users/rkm/beeware/briefcase/src\briefcase\__main__.py'.
This problem still exists if you use relative_files
; you just don't need the prefixes
entry. However, as long as you've got at least one directory in your project structure the path separators don't match, and you're doing a parallel run that includes both a Windows machine and a Unix/macOS machine, you'll hit this problem.
The naïve fix is to make all path separators "unix" style when they're persisted in the coverage file - which is what the SQLite hack does. There might be other problems with taking that approach (especially if you're running the report on Windows) - but it's been enough to get me unstuck.
@freakboy3742 are you using the [paths]
setting to help combine files from different machines? I didn't see it in your branch for https://github.com/beeware/briefcase/pull/887. When combining, we try to adjust the separators so they will work properly across operating systems. Perhaps that's a non-obvious approach.
What's the best way for me to see the failing case when the separators aren't adjusted?
@nedbat I'm not; I'm using relative_paths
because the base paths aren't predictable (because end users and/or CI won't be using C:\Users\rkm\beeware\briefcase
as the source location).
As for the failing case - the coverage files I've uploaded contain the generated data; that data is from commit 0cd2ca0, CI run https://github.com/beeware/briefcase/actions/runs/3155776011.
I've made some changes on a branch:
pip install git+https://github.com/nedbat/coveragepy@nedbat/bug991
@cjolowicz This should make your code combine and report correctly as-is. @freakboy3742 Can you add this to your pyproject.toml and try it without your SQLite fixup:?
[tool.coverage.paths]
src = ["src", "*/src"]
Perhaps this should be more automatic, since src = ["src", "*/src"]
seems kind of self-evident in a way.
@nedbat Can confirm the branch plus that config change fixes the problem I'm seeing without the need for the SQLite fix.
Thanks for taking a look at this again nedbat. I tried out the branch and also added a trivial path section to the config and I still get errors around the \
path separator. https://github.com/Chia-Network/chia-blockchain/pull/13587 This seems to be counter to comments above so perhaps I am still missing a piece.
https://github.com/Chia-Network/chia-blockchain/actions/runs/3175559386/jobs/5173910237#step:7:66
+ coverage xml --rcfile=.coveragerc --data-file coverage-reports/.coverage -o coverage-reports/coverage.xml
No source for code: '/home/runner/work/chia-blockchain/chia-blockchain/chia\__init__.py'.
Here is the [paths]
configuration I added.
https://github.com/Chia-Network/chia-blockchain/blob/test_coverage_991/.coveragerc#L10-L12
[paths]
chia = ["chia", "*/chia"]
tests = ["tests", "*/tests"]
I added some diagnostics as reference against the error message. https://github.com/Chia-Network/chia-blockchain/actions/runs/3175559386/jobs/5173910237#step:7:15 (note that I have slightly edited this to compensate for the command output streams not being synced to the output exactly
ls -la '/home/runner/work/chia-blockchain/chia-blockchain/chia\__init__.py'
+ ls -la '/home/runner/work/chia-blockchain/chia-blockchain/chia\__init__.py'
ls: cannot access '/home/runner/work/chia-blockchain/chia-blockchain/chia\__init__.py': No such file or directory
ls -la '/home/runner/work/chia-blockchain/chia-blockchain/chia/__init__.py'
+ ls -la /home/runner/work/chia-blockchain/chia-blockchain/chia/__init__.py
-rw-r--r-- 1 runner docker 347 Oct 3 16:03 /home/runner/work/chia-blockchain/chia-blockchain/chia/__init__.py
ls -la '/home/runner/work/chia-blockchain/chia-blockchain/chia'
+ ls -la /home/runner/work/chia-blockchain/chia-blockchain/chia
total 112
drwxr-xr-x 25 runner docker 4096 Oct 3 16:03 .
drwxr-xr-x 14 runner docker 4096 Oct 3 16:04 ..
-rw-r--r-- 1 runner docker 347 Oct 3 16:03 __init__.py
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 clvm
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 cmds
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 consensus
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 daemon
drwxr-xr-x 3 runner docker 4096 Oct 3 16:03 data_layer
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 farmer
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 full_node
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 harvester
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 introducer
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 plot_sync
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 plotters
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 plotting
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 pools
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 protocols
-rw-r--r-- 1 runner docker 0 Oct 3 16:03 py.typed
-rw-r--r-- 1 runner docker 5514 Oct 3 16:03 pyinstaller.spec
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 rpc
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 seeder
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 server
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 simulator
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 ssl
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 timelord
drwxr-xr-x 3 runner docker 4096 Oct 3 16:03 types
drwxr-xr-x 2 runner docker 4096 Oct 3 16:03 util
drwxr-xr-x 10 runner docker 4096 Oct 3 16:03 wallet
@altendky thanks for looking into it. Can you add --debug=pathmap
to the coverage combine
step? It will show the file names it is considering, and how the get remapped.
Sure thing.
https://github.com/Chia-Network/chia-blockchain/actions/runs/3177768807/jobs/5178689238#step:7:54
coverage combine --rcfile=.coveragerc --debug=pathmap --data-file coverage-reports/.coverage coverage-data/
+ coverage combine --rcfile=.coveragerc --debug=pathmap --data-file coverage-reports/.coverage coverage-data/
Aliases (relative=True):
Rule: '"*/chia"]' -> '["chia"/' using regex '(?s:[\\\\/]home[\\\\/]runner[\\\\/]work[\\\\/]chia\\-blockchain[\\\\/]chia\\-blockchain[\\\\/]".*[\\\\/]chia"\\][\\\\/])'
Rule: '"*/tests"]' -> '["tests"/' using regex '(?s:[\\\\/]home[\\\\/]runner[\\\\/]work[\\\\/]chia\\-blockchain[\\\\/]chia\\-blockchain[\\\\/]".*[\\\\/]tests"\\][\\\\/])'
No rules match, path 'tests/__init__.py' is unchanged
No rules match, path 'tests/conftest.py' is unchanged
No rules match, path 'tests/setup_nodes.py' is unchanged
No rules match, path 'chia/__init__.py' is unchanged
No rules match, path 'chia/consensus/__init__.py' is unchanged
No rules match, path 'chia/consensus/constants.py' is unchanged
No rules match, path 'chia/types/__init__.py' is unchanged
<snip>
No rules match, path 'tests\\wallet\\sync\\config.py' is unchanged
No rules match, path 'tests\\wallet\\sync\\test_wallet_sync.py' is unchanged
No rules match, path 'tests\\wallet\\sync\\__init__.py' is unchanged
No rules match, path 'tests\\weight_proof\\config.py' is unchanged
No rules match, path 'tests\\weight_proof\\test_weight_proof.py' is unchanged
No rules match, path 'tests\\weight_proof\\__init__.py' is unchanged
Combined data file coverage-data/.coverage.tests_ubuntu_python-3.10_core.cmds
Combined data file coverage-data/.coverage.tests_ubuntu_python-3.7_core.cmds
Combined data file coverage-data/.coverage.tests_ubuntu_python-3.9_core.cmds
Combined data file coverage-data/.coverage.tests_windows_python-3.10_core.cmds
Combined data file coverage-data/.coverage.tests_macos_python-3.9_core.cmds
Combined data file coverage-data/.coverage.tests_macos_python-3.10_core.cmds
Combined data file coverage-data/.coverage.tests_ubuntu_python-3.8_core.cmds
Combined data file coverage-data/.coverage.tests_windows_python-3.7_core.cmds
Combined data file coverage-data/.coverage.tests_windows_python-3.9_core.cmds
Combined data file coverage-data/.coverage.tests_windows_python-3.8_core.cmds
@altendky I see what happened. The earlier discussion was about a pyproject.toml file, and you put that snippet into your .coveragerc file. The syntax you need is different:
[paths]
chia =
chia
*/chia
tests =
tests
*/tests
And here I thought I was being a good little dev and exactly following the example. :]
Fixing now.
Yep, that works. Thanks for identifying my error.
https://github.com/Chia-Network/chia-blockchain/actions/runs/3177937330/jobs/5179030056#step:7:3405
Matched path 'chia\\__init__.py' to rule '*/chia' -> 'chia/', producing 'chia/__init__.py'
It is apparently not obvious that this would be needed to trigger platform compatibility. :]
I would suggest considering making directory separator compatibility automatic and perhaps separate from path equivalency configuration.
But anyways, looks like we'll be turning on Windows coverage inclusion 'soon' and I very much appreciate that. Now, to take the next step and get this data well reported and require coverage for PRs...
Hi, I've added more to the nedbat/bug991 branch (commit 285ca254). Now relative paths will be combined implicitly, so you shouldn't need "obvious" [paths]
configurations anymore. @freakboy3742 @altendky can you try it on your projects with no [paths]
settings at all? You will still need the coverage combine
step.
@nedbat No luck, sorry - commit 285ca25 without a [paths]
config isn't working for me.
[tool.coverage.run]
parallel = true
branch = true
relative_files = true
source = ["src/briefcase"]
[tool.coverage.report]
show_missing = true
skip_covered = true
skip_empty = true
precision = 1
exclude_lines = [
"pragma: no cover",
"@(abc\\.)?abstractmethod",
"NotImplementedError\\(\\)"
]
(venv3.10) rkm@stirtoni briefcase % python -m coverage combine
Combined data file .coverage.stirtoni.local.21166.405507
Combined data file .coverage.RUSSELLKEITA152.6268.549117
(venv3.10) rkm@stirtoni briefcase % python -m coverage report
No source for code: '/Users/rkm/beeware/briefcase/src\briefcase\__init__.py'.
I get the same result if I disable relative_paths
, except the error becomes:
No source for code: '/Users/rkm/beeware/briefcase/\\MAC\HOME\beeware\briefcase\src\briefcase\__init__.py'.
@freakboy3742 hmm, i've been running your project as a test, that's odd. Can you try these commands?
python -m coverage debug data
python -m coverage combine --debug=pathmap
python -m coverage debug data
Seems ok here without the [paths]
section.
https://github.com/Chia-Network/chia-blockchain/actions/runs/3216374294/jobs/5258265284#step:7:3472
Matched path 'chia\\__init__.py' to rule '*/chia' -> 'chia/', producing 'chia/__init__.py'
Also looks ok in the next run where I removed the coverage debug, other debug, and CI minimization.
Thanks for cleaning this up here. :]
@nedbat Log output below.
Test suite is running on an x86 Mac mini running Monterey (12.5.1), and a Parallels virtual machine on the same physical machine running Windows 10. The Parallels machine is accessing the source folder over a network drive (hence the \\MAC\HOME\beeware
path mentioned in my last post). The Briefcase repo is at commit 385a4aa1.
@freakboy3742 thanks for sticking with me on this. I don't understand why the debug output isn't appearing in the combine step. Can you make it --debug=sys,config,pathmap
, and we'll get some clues? It seems like it's not running the code from the branch, but I see that it installed the code from the branch.
@nedbat
@freakboy3742 I don't understand why there's no output from the combine step. I've added a more distinctive version marker to the branch. Can you try running with nedbat/bug991 again?
Ok - that seems to have worked. Running on Briefcase hash 385a4aa1:
I'm marking this as fixed as of commit 7df8609f.
But I think because of #1407, there will be larger changes to how file wildcards are interpreted.
This is now released as part of coverage 6.6.0b1.
This is now released as part of coverage 7.0.0b1.
Current Problem I'm running coverage for multiple Python versions on both Linux and Windows. They are all run on Gitlab, with the Windows tests using Python install on the host, and with the Linux tests using Docker. The paths of the build directories are not deterministic (may be picked up by multiple runners, existing or yet to exist) so I have
relative_files = True
.The Windows paths use
\
instead of/
which, if the coverage combination and reporting is done on Windows, results in a report with every source file twice, one for each separator, although the end result appears to be correct. If this is done on Linux, it fails entirely when trying to resolve the paths containing backslashes.Example Report
Describe the solution you'd like Source paths on Windows should use forward slashes in relative paths to allow result combination and final reporting to be done on Linux, and to clean up the report output when done on Windows.
Describe alternatives you've considered Manually modifying the coverage results generated from
coverage run
.