Figure out how to run code-coverage against a component and get some realistic results.

[data-sync-user]

┆Issue is synchronized with this Jira Spike ┆Story Points: 5 ┆Epic: a-s test coverage ┆Sprint: SYNC - end 2020-04-24

[data-sync-user]

➤ Ryan Kelly commented:

Back in https://jira.mozilla.com/browse/SYNC-683 we did a little experiment with using cargo-tarpaulin ( https://github.com/xd009642/tarpaulin ) to measure test coverage for the logins component. It worked OK but reported low coverage, because much of the test coverage for that code comes from places other than cargo test, such as:

• The Kotlin tests, which exercise the rust code through the FFI layer
• The sync integration tests in ./testing/sync-test, which exercise sync-related code by doing some live tests

The aim of this spike is to figure out how we can run all of these tests and get a realistic code-coverage report for a component. It doesn't have to be the logins component, but that seems to be one of our better-tested ones, so it's probably a good place to start.

The glean team have some notes on how they do code coverage here:

https://mozilla.github.io/glean/book/code_coverage.html#generating-rust-reports-locally

IIUC this basically involves:

• compiling the rust code with special flags to tell it to emit code-coverage data using llvm's native coverage support
• running the rust code
• using grcov ( https://github.com/mozilla/grcov ) to aggregate the generated data and pipe it to other tools for reporting/display

I suspect, but have not confirmed, that we could use this approach to get a more comprehensive coverage report. We could compile the megazord with the special coverage-related flags, run cargo-test against it to generate coverage data for the rust-level tests, run the kotlin-level tests against that specially-compiled megazord to generate coverage data for the kotlin-level tests, then aggregate the results into a single report.

[vladikoff]

At this time the custom RUSTFLAGS required by grcov make the nss build to fail:

error: failed to run custom build command for `nss_sys v0.1.0 (/Users/vladikoff4/mozilla/application-services/components/support/rc_crypto/nss/nss_sys)`

Caused by:
  process didn't exit successfully: `/Users/vladikoff4/mozilla/application-services/target/debug/build/nss_sys-9ecf6a110e079ba1/build-script-build` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=NSS_DIR
cargo:rustc-link-search=native=/Users/vladikoff4/mozilla/application-services/libs/desktop/darwin/nss/lib
cargo:include=/Users/vladikoff4/mozilla/application-services/libs/desktop/darwin/nss/include
cargo:rerun-if-env-changed=NSS_STATIC
cargo:rustc-link-lib=static=certdb
cargo:rustc-link-lib=static=certhi
cargo:rustc-link-lib=static=cryptohi
cargo:rustc-link-lib=static=freebl_static
cargo:rustc-link-lib=static=hw-acc-crypto
cargo:rustc-link-lib=static=nspr4
cargo:rustc-link-lib=static=nss_static
cargo:rustc-link-lib=static=nssb
cargo:rustc-link-lib=static=nssdev
cargo:rustc-link-lib=static=nsspki
cargo:rustc-link-lib=static=nssutil
cargo:rustc-link-lib=static=pk11wrap_static
cargo:rustc-link-lib=static=plc4
cargo:rustc-link-lib=static=plds4
cargo:rustc-link-lib=static=softokn_static
cargo:rustc-link-lib=static=gcm-aes-x86_c_lib
cargo:rerun-if-changed=bindings.toml
cargo:warning=couldn't execute `llvm-config --prefix` (error: No such file or directory (os error 2))
cargo:warning=set the LLVM_CONFIG_PATH environment variable to the full path to a valid `llvm-config` executable (including the executable itself)

Not sure what the cause of that is just yet

[vladikoff]

re: using grcov on the currently would fail with the following STR for nss:

cd /components/support/rc_crypto/nss
export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads"
cargo +nightly  test 

might just use tarpaulin even though it is linux only

[thomcc]

The issue with tarpaulin wasn't that it's linux only (although this is true) so much as the fact that it failed to be able to give us coverage information for stuff tested via kotlin

[vladikoff]

The issue with tarpaulin wasn't that it's linux only (although this is true) so much as the fact that it failed to be able to give us coverage information for stuff tested via kotlin

Yeah, one solution I've looked into was to combined the lcov report from different sources! (rust lcov + kotlin lvoc + etc) That possibly produce a full report of the test coverage.

[vladikoff]

Observation, for unknown to me reason, sync-guid failed to run through cargo tarpaulin for me. Trying with isolated components

[data-sync-user]

➤ Vladislav Filippov commented:

After looking into various tools, It seems tarpauilin was the only one able to produce good results in my case. I have filed a few issues in this epic to break down the work we can do to add code coverage to the project. E.g: https://jira.mozilla.com/browse/SYNC-1165