search

taiki-e / cargo-llvm-cov

Cargo subcommand to easily use LLVM source-based code coverage (-C instrument-coverage).
Apache License 2.0
933 stars 57 forks source link

Usage in CI #12

Closed taiki-e closed 3 years ago

taiki-e commented 3 years ago

codecov claims to support many formats, but it actually happens "There was an error processing coverage reports" errors on the server. It is necessary to understand which formats are actually supported and which ones are preferred to use.

I'm not sure about other code-coverage services as I have never used them.

jhpratt commented 3 years ago

Check out the time-rs/time repo. I just added codecov using json successfully. lcov also worked.

jhpratt commented 3 years ago

Actually I stand corrected. JSON does not work. lcov does. https://app.codecov.io/gh/time-rs/time

taiki-e commented 3 years ago

Thanks @jhpratt. I also checked it in my test repository, it seems to work only with lcov. (Uploading of json and text reports succeeded, but an error happened in the server.)

jhpratt commented 3 years ago

No problem. It looks like codecov only cares about line coverage as well, rather than the more detailed report lcov provides. But I'm pretty sure that's on codecov's end.

taiki-e commented 3 years ago

It looks like codecov only cares about line coverage as well, rather than the more detailed report lcov provides. But I'm pretty sure that's on codecov's end.

Hmm. It seems codecov supports branch coverage, so that might be a rustc's problem (#8, https://github.com/rust-lang/rust/issues/79649).

https://docs.codecov.io/docs/about-code-coverage

partial indicates that the source code was not fully executed by the test suite; there are remaining branches that were not executed.

llvm-cov (that used by cargo-llvm-cov internally) also supports the export of branch coverage.

https://llvm.org/docs/CommandGuide/llvm-cov.html#llvm-cov-export

When exporting an lcov trace file, the line-based coverage, branch coverage, and summaries will be exported.

taiki-e commented 3 years ago

It looks like codecov only cares about line coverage as well, rather than the more detailed report lcov provides. But I'm pretty sure that's on codecov's end.

We can probably support region coverage on codecov by converting llvm-cov's json coverage format to codecov's json coverage format.