Open ulfjack opened 4 years ago
In order for cross-language coverage to work in Bazel, all coverage data needs to be converted into a common format. The above workaround uses an upstream patch to coverage.py. As an alternative, we could also change LcovMerger to support coverage.py's output format.
Reference to upstream's issue: https://github.com/nedbat/coveragepy/issues/587
The issue you were looking for may have been under python_rules https://github.com/bazelbuild/rules_python/issues/43
Thanks for the suggested workaround!
@ulfjack thank you for the suggestion. Unfortunately I am seeing empty coverage.dat file with some warnings in in the test.log
file. Any idea what I might be missing?
test.log
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: No lcov file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main main
WARNING: There was no coverage found.
Also is there a documentation about using Bazel Coverage on python that might be helpful to read?
The most likely reason for coverage not to work is an incorrect --instrumentation_filter
, and the second most likely is having sources in a *_test
rule - test rules do not collect any coverage. Beyond that, there are a number of things that could go wrong, but I don't know which one it is. You'd have to post a repro if you need more help.
@ulfjack I get empty coverage files regardless what I try, and these are repositories with only __init__.py
and test_...py
files in the srcs
attribute. The code under test in one project are in libraries and referenced as dependencies (ie not in srcs
, but also not an embed
of any kind).
Do you have a basic example that you used during development? Or a test case that works? It would be really helpful to see any sort of functioning test to fast-track some experimentation.
For reference, I've tried various combinations of this script, changing the parameters of the last line to get a run that doesn't fail, but gives non-zero coverage.dat files. So far, never ever a nonzero coverage.dat.
#!bash
COVERAGEPY=${HOME}/src/coveragepy
test -d "${COVERAGEPY}" || git clone -b lcov-support https://github.com/ulfjack/coveragepy.git "${COVERAGEPY}"
# According to https://github.com/bazelbuild/bazel/issues/10660, and some assumption:
bazel coverage --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py //:test_ns
# Now, in other documentation, there is much discussion about local_jdk and various coverage flags.
# The following completes without error, but still empty coverage.dat files
#
# NOTE: //:test_ns is the only test target for this example.
bazel coverage -s --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py --sandbox_debug \
--instrument_test_targets --instrumentation_filter=//...:all --combined_report=lcov \
--javabase=@bazel_tools//tools/jdk:remote_jdk11 //:test_ns
Sometimes I also try with --coverage_report_generator= @bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main
but there seems no difference.
@chickenandpork I've been generating reports for a while now. Not directly with bazel, but by manually calling grcov
in conjunction with coveragepy. Here's a sample from one of my private repo actions:
- name: Python coverage
run: |
cd "${GITHUB_WORKSPACE}/src"
curl -L https://github.com/ulfjack/coveragepy/archive/lcov-support.tar.gz | tar xvz
~/bin/bazel coverage -t- --instrument_test_targets --experimental_cc_coverage \
--test_output=errors --linkopt=--coverage --linkopt=-lc \
--test_env=PYTHON_COVERAGE=${GITHUB_WORKSPACE}/src/coveragepy-lcov-support/__main__.py \
--define=config_file=test //python/:tests
# Unfortunately, coverage does not produce the hit information. Running
# the compiled tests explicitly seems to work though.
find bazel-bin/python/ -maxdepth 1 -regex ".*-test$" -exec {} \;
cp bazel-out/k8-fastbuild/testlogs/pd3/python/smoke-test/coverage.dat coverage.py.info
env:
CC: clang-9
- name: codecov
if: always()
run: |
set -x
cd "${GITHUB_WORKSPACE}/src"
~/bin/grcov coverage.py.info bazel-bin/python/_objs/ -t lcov \
--ignore "external/*" --ignore "/usr/*" \
--ignore "*deps_*" --ignore "*_pb2.py" \
--llvm > lcov.py.info
sed -i 's/SF:.*test\.runfiles\/python\/python/SF:python/g' coverage.py.info
cd ..
bash -x <(curl -s https://codecov.io/bash) -Z -v -f src/lcov.py.info -F python_tests
env:
CC: clang-9
CODECOV_TOKEN: ${{ secrets.CODE_COV }}
/cc @oquenchil
Maybe this can fit into your coverage sprint?
@chickenandpork In case this is still relevant for you, or maybe this will help someone:
bazel's --instrumentation_filter
option is a regular expression, so passing --instrumentation_filter=//...:all
will not have the effect of instrumenting everything, as one would expect. Instead likely it will not instrument anything.
Leaving the option unset, bazel will generally guess a good value, or if you want to instrument everything all the time, then --instrumentation_filter=//
might do what you intended.
@chickenandpork @ulfjack If you use a pytest_runner to run your pytest tests, it's a convenient place to put the lconv conversion code ;) That way you can still use standard tools (and not need to use the fork or special shell scripts)
The following requires pytest-cov
andcoverage-lconv
import contextlib
import os
import pathlib
import sys
import coverage
import coverage_lcov.converter
import pytest
@contextlib.contextmanager
def coverage_decorator():
if os.getenv("COVERAGE", None) == "1":
coverage_dir = pathlib.Path(os.getenv("COVERAGE_DIR"))
coverage_file = coverage_dir / ".coverage"
coverage_manifest = pathlib.Path(os.getenv("COVERAGE_MANIFEST"))
coverage_sources = coverage_manifest.read_text().splitlines()
# @TODO: Handle config stuff
cov = coverage.Coverage(data_file=str(coverage_file), include=coverage_sources)
cov.start()
try:
yield
finally:
if os.getenv("COVERAGE", None) == "1":
cov.stop()
cov.save()
# @TODO: Handle config stuff
coverage_lcov.converter.Converter(
relative_path=True,
config_file=False,
data_file_path=str(coverage_file),
).create_lcov(os.getenv("COVERAGE_OUTPUT_FILE"))
if __name__ == "__main__":
with coverage_decorator():
sys.exit(pytest.main(sys.argv[1:]))
The dropbox Python rules also seem to do something similar for coverage support: https://github.com/dropbox/dbx_build_tools/blob/fe5c9e668a9e6951970c0595089742d8a0247b8c/build_tools/py/pytest_plugins/codecoverage.py
py_test
for pybind module does not work either. I expected it to work since the module is a cc_binary
and hence gcov/lcov should be sufficient. All bazel commands - coverage|test|run <py-test-target>
- produce empty coverage report.
Interestingly running the py-test-target
directly produces .gcda files as expected. So it seems bazel is simply ignoring the .gcda files for py_test targets? Has anyone seen this before and knows of a workaround?
FYI, I created https://github.com/bazelbuild/rules_python/pull/977 to make bazel coverage
work with the hermetic Python toolchain from @rules_python
. I found some issues whilst doing the work and I'll copy them here for visibility. The issues were:
coverage.py
v6.5.0 because the latest version (7.0.4
has a types.py
file in the package directory, which imports from Python's stdlib types [1]. Somehow the Python interpreter is thinking that the from types import FrameType
is referring to the currently interpreted file and everything breaks. I would have expected the package to use absolute imports and only attempt to import from coverage.types
if we use coverage.types
and not just a plain types
import.multi_python_versions
example cannot show coverage for the more complex tests that are using subprocess
to spawn a different Python interpreter. I am wondering if this is related to the fact that we are including coverage.py
via the toolchain and not through other mechanisms [2].__init__.py
files in the root of the bzlmod
example was breaking, when running under bazel coverage //:test. However, it started working when I renamed __init__.py
to lib.py
. I am suspecting that this has to do with the fact that the layer of indirection that coverage introduces could be something to do with that. Note that bazel test ...
works regardless of file naming.I think that all of these issues may be related to Python entrypoint template that is stored in this repository and not an issue with rules_python
itself.
I'll admit to not having actually checked any test cases for a __init__.py
module. TBH python's import processing still confuses me in some cases, especially around virtual packages. It's possible that the template is doing something wrong in that case. However, what did you have set for --incompatible_default_to_explicit_init_py? Because I could definitely see that messing things up.
And yes, you won't get coverage in subprocesses, because coverage
works by installing its hooks into the interpreter process your code is running in, and it isn't going to install those in a subprocess. Not sure if there's a good way around that short of hacky attempts to detect when the process is running with coverage enabled, which seems like a bad idea.
@adam-azarchs there is a pytest plugin called pytest-cov that is able to get around this issue and will profile all subprocesses. this means that it will also work with pytest-xdist which is not compatible with bazel coverage
right now. a call to coverage combine
after coverage run
and before coverage lcov
I think should at least allow using this common plugin
Last I checked, though, pytest-cov
couldn't output in LCOV format, which is a requirement.
(also pytest-cov
requires pytest
, which is a pretty heavy-weight dependency)
Ah sorry I was a bit confusing - I meant that it would allow using pytest-xdist which is quite common.
Pytest-cov is now able to output in lcov format just as coverage can. And you're absolutely right about pytest bring heavy.
What I really meant to suggest is that between pytest-cov and coverage-enable-subprocess there's probably enough info to make bazel compatible with these subprocesses during coverage automatically and natively if the desire is there. But adding a call to coverage combine
would at least allow people to create their own workarounds to the problem. The combine command doesn't do anything if there are not multiple files to combine so would be safe for existing users
Hi All, Has anyone in this thread come across any interesting workarounds and solutions to the original issue? I can't decide if I should dive into the various workarounds suggested already or perhaps waiting a bit longer for a permanent fix/workaround to the issue.
rules_python now supports coverage natively: https://rules-python.readthedocs.io/en/stable/coverage.html Does that work for you?
Hi @phst Coverage works however the coverage reports are empty so it's not possible to produce a merged coverage report across test targets.
I thought that there was an existing issue for this, but I can't find it.
I was able to successfully collect coverage for python code with a modified version of coverage.py:
Test coverage should be written to
bazel-testlogs/python/lib_test/coverage.dat
in lcov format.