Generate coverage output for Gitlab/Cobertura?

[TheButlah]

Would it be possible to add support for generating Cobertura xml coverage reports? Gitlab's coverage reporting uses Cobertura's format.

It appears as though tarpaulin supports cobertura, so it would be good if grcov could too.

[marco-c]

Sure, I won't have time to implement it myself in the near term, but I'd be happy to review a PR to add support for it.

[brycefisher]

@TheButlah I just ran into the same thing myself! There is actually this Python project that converts from lcov to cobertura: https://github.com/eriwen/lcov-to-cobertura-xml

Since I'm using the (un"slim") official Rust docker image on GitlabCI, Python 3 is already available. Right after running grcov, I added these lines in the script block, and cobertura seemed to work:

    - curl -O https://raw.githubusercontent.com/eriwen/lcov-to-cobertura-xml/master/lcov_cobertura/lcov_cobertura.py && chmod +x lcov_cobertura.py
    - ./lcov_cobertura.py lcov.info

This solves my needs for now, but is obviously less efficient than simply generating cobertura directly in grcov.

[royb3]

@brycefisher Does it work fine with Gitlab CI? What about function name mangling?

[brycefisher]

It does!! It works great. You can see test code coverage within the merge request UI...kind of awesome!

Not my ideal solution (and adds a dependency to use Python for testing an otherwise pure Rust project), but its a decent workaround for the moment.

RE: function name mangling, I've only worked with pure Rust -- no extern "C" style FFI or anything. Do you have a good test case in mind that would verify if name mangling works out of the box?

[gdesmott]

Thanks a lot @brycefisher that works great!

By any chance, did you find a way to also integrate the coverage summary, see #556 ?

[brycefisher]

script:
    - ...
    - grcov ./grcov.zip -s . --llvm --branch --ignore-not-existing -o ./lcov.info
    - if [[ ! -e lcov_cobertura.py ]]; then curl -O https://raw.githubusercontent.com/eriwen/lcov-to-cobertura-xml/master/lcov_cobertura/lcov_cobertura.py && chmod +x lcov_cobertura.py; fi
    - ./lcov_cobertura.py lcov.info
    - grep '<coverage branch-rate="' coverage.xml | sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'

[seanpianka]

@brycefisher When I use the python (3) script on my lcov.info output, I get this error message...

$ grcov "${ZIP_NAME}" \
    --source-dir . \
    --binary-path ./target/debug/ \
    --output-path "${COVERAGE_OUTPUT_PATH}" \
    --branch \
    --llvm \
    --ignore-not-existing \
    --keep-only "mycoolcrate-*" 
$ scripts/lcov_cobertura.py "${COVERAGE_OUTPUT_PATH}"
Traceback (most recent call last):
  File "scripts/lcov_cobertura.py", line 416, in <module>
    main()
  File "scripts/lcov_cobertura.py", line 409, in main
    cobertura_xml = lcov_cobertura.convert()
  File "scripts/lcov_cobertura.py", line 87, in convert
    coverage_data = self.parse()
  File "scripts/lcov_cobertura.py", line 197, in parse
    function_line, function_name = line_parts[-1].strip().split(',')
ValueError: too many values to unpack (expected 2)

Any advice on what could be wrong here? Thanks.

Edit: Here's a fix... only splitting once for FN and FNDA:

diff --git a/scripts/lcov_cobertura.py b/scripts/lcov_cobertura.py
index affef27a..0c07b2af 100755
--- a/scripts/lcov_cobertura.py
+++ b/scripts/lcov_cobertura.py
@@ -194,11 +194,11 @@ class LcovCobertura(object):
                 file_branches_covered = int(line_parts[1])
             elif input_type == 'FN':
                 # FN:5,(anonymous_1)
-                function_line, function_name = line_parts[-1].strip().split(',')
+                function_line, function_name = line_parts[-1].strip().split(',', 1)
                 file_methods[function_name] = [function_line, '0']
             elif input_type == 'FNDA':
                 # FNDA:0,(anonymous_1)
-                (function_hits, function_name) = line_parts[-1].strip().split(',')
+                (function_hits, function_name) = line_parts[-1].strip().split(',', 1)
                 if function_name not in file_methods:
                     file_methods[function_name] = ['0', '0']
                 file_methods[function_name][-1] = function_hits

[brycefisher]

Cool! My original snippet pulled from master on the Python project, so either it never worked for your use case, or an update since I posted broke it, or something along those lines. Sometimes, pinning a version can be handy for making things continue to work the same way over time (that or making you into a bitrotten purgatory that never ends...YMMV)

[seanpianka]

Yep, thanks for mentioning that conversion script! Could you help with an issue related to GitLab?

I export the generated coverage.xml report, but GitLab seems to do nothing with it... I don't see anything related to code-coverage in the merge request, no results or line highlighting.

all:
  extends:
    - .unit-test
  script:
    # Generate Cobertura report
    - grcov "${COVERAGE_ZIP_NAME}" \
        --source-dir . \
        --binary-path ./target/debug/ \
        --output-path "${COVERAGE_OUTPUT_PATH}/lcov.info" \
        --output-type lcov \
        --branch \
        --llvm \
        --ignore-not-existing \
        --keep-only "mycoolcrate-*"
    # Use a cargo extension to correct the code coverage results... removes extranous counts.
    - rust-covfix -o "${COVERAGE_OUTPUT_PATH}/lcov.info" "${COVERAGE_OUTPUT_PATH}/lcov.info"
    # outputs cobertura report at "./coverage.xml"
    - python3 scripts/lcov_cobertura.py "${COVERAGE_OUTPUT_PATH}/lcov.info"
  artifacts:
    reports:
      cobertura: coverage.xml

Do I need to use the grep command you posted, and use a regex to parse the code coverage from the job output?

$ grep '<coverage branch-rate="' coverage.xml | sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'

[seanpianka]

Also, is there anyway to use an lcov.info file as input to grcov -t html? I don't want grcov to regenerate the info file, I want it to re-use my processed version (the output from rust-covfix)... I can open a separate issue, if needed.

[brycefisher]

Okay, so there's two parts:

1. Feeding the full cobertura XML data into Gitlab so it can show inline test coverage on MRs. For that, the path to the coverage.xml file must be correct in the artifacts:reports:cobertura section of your .gitlabci.yml. I suspect that you might have feed the wrong path there? I generally start throwing ls and cat statements into my CI, or I download and run the gitlab runner locally to debug faster (https://bryce.fisher-fleig.org/faster-ci-debugging-with-gitlabci/ -- quite old, possibly outdated!)
2. Getting an overall percentage into a badge you can display on your README. That's what the grep bit does you quoted

[brycefisher]

Oh! One more thing, i think that you also have to have a baseline from the target branch before this stuff starts being helpful. So maybe merge a trivial branch to master, or do a MR against a feature branch. Can't remember where the docs spell that out on GitlabCI....

[seanpianka]

You're right, the code coverage results are visible in the merge requests! I completely overlooked those green/red lines next to each line... thanks for the tip! 😄

Uploading artifacts for successful job
Uploading artifacts...
html-coverage-report: found 503 matching files and directories 
artifact_job_id: found 1 matching files and directories 
Uploading artifacts as "archive" to coordinator... ok  id=108... responseStatus=201 Created token=5sym...
Uploading artifacts...
coverage.xml: found 1 matching files and directories 
Uploading artifacts as "cobertura" to coordinator... ok  id=108... responseStatus=201 Created token=5sym...

So, for getting the code coverage result for the badge (or somehow visible on the "Overview" page of each merge request?), I have to run the quoted grep command, then add a parsing regex to find it in the job output.

What's the parsing regex you used in your project settings? Do you think that Test coverage \d+.\d+% would work?

[brycefisher]

Yeah, here what I have: Test coverage (\d+\.\d)%

[seanpianka]

Using the quoted grep, I see a peculiar output:

+ python3 scripts/lcov_cobertura.py target/debug/coverage/lcov.info
+ grep '<coverage branch-rate="' coverage.xml
+ sed -e 's/.* branch-rate="\(.\)\.\(..\)\(.\).*/Test coverage \1\2.\3%/'
Test coverage 011.2%

Perhaps one of the capture groups is unnecessary?

[seanpianka]

This seems to work alright!

# raw branch-rate value
local branch_rate
branch_rate=$(grep '<coverage branch-rate="\(.*\)"' coverage.xml | sed -e 's/.* branch-rate="\(.*\)" branches-covered.*/\1/')

# multiply by 100 with bash syntax
local coverage
coverage=$(printf %.2f "$(echo "$branch_rate * 100" | bc -l)")

# format with %
echo "Test coverage ${coverage}%"

[geovie]

FYI: https://github.com/mozilla/grcov/pull/599 tries to implement this for grcov.

Please give it a try 🙏

[marco-c]

Cobertura output was implemented, it'd be great if one of you could add some docs in README.md to explain how to use it in GitLab (just like the "grcov with Travis" section).