search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.58k stars 12.74k forks source link

Regression in code coverage (`-Zprofile`) at some point after `nightly-2021-11-11` #100125

Closed fadeevab closed 1 week ago

fadeevab commented 2 years ago

TL;DR:

Older nightly toolchain gives an expected code coverage: rustup default nightly-2021-11-11

UPDATE: Regression is reproduced below https://github.com/rust-lang/rust/issues/100125#issuecomment-1209535289

Prerequisites

I have a GitHub workflow which generates a code coverage report with nightly toolchain +grcov, then sending it to https://coveralls.io.

export CARGO_INCREMENTAL=0
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
export RUSTDOCFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"

rustup default nightly
cargo test --all-features
curl -L https://github.com/mozilla/grcov/releases/latest/download/grcov-x86_64-unknown-linux-gnu.tar.bz2 | tar jxf -

mkdir coverage
./grcov ./target/debug/ -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" \
                --excl-line '#\[|=> panic!|unreachable!|Io\(std::io::Error\)' \
                --excl-br-line '#\[|=> panic!|unreachable!|assert_..!' -o ./coverage/lcov.info

I expected to see 100% code coverage on a simple file, e.g. see older coverage result: https://coveralls.io/builds/43003664/source?filename=src%2Ferror.rs - error.rs has 100% code coverage.

Instead, this happened:

  1. https://coveralls.io/builds/51398291/source?filename=src%2Ferror.rs - 9% coverage.
  2. https://coveralls.io/builds/51398291/source?filename=src%2Flib.rs - coverage goes over comment lines

Version it worked on

At least nightly-2021-11-11 toolchain works:

rustup default nightly-2021-11-11

image

However, I have not narrowed down the most recent version this worked on.

Version with regression

rustc --version --verbose:

rustc 1.64.0-nightly (0f4bcadb4 2022-07-30)
binary: rustc
commit-hash: 0f4bcadb46006bc484dad85616b484f93879ca4e
commit-date: 2022-07-30
host: x86_64-unknown-linux-gnu
release: 1.64.0-nightly
LLVM version: 14.0.6
apiraino commented 2 years ago

Unless I am mistaken, grcov is a third-party tool, not officially under the Rust project umbrella.

In order to find an actionable here, a bisection and ideally a reproducible are probably the way to go.

@rustbot label E-needs-bisection E-needs-mcve

fadeevab commented 2 years ago

@apiraino I've spent a few hours yesterday, and that's my result:

  1. latest grcov + latest rust nightly = bad
  2. old grcov 0.8.2 (~July 2021) + latest rustc nightly = bad
  3. old grcov + old rustc nightly-2021-11-11 = good
  4. latest grcov + old nightly-2021-11-11 = good

Notes:

Case 1 and Case 2 above shows, that the newer nightly either has a regression, or the nightly introduces some profile format incompatibility with 3rd party tools.

fanninpm commented 2 years ago

You can help narrow down the regression even more by using cargo-bisect-rustc. The cargo-bisect-rustc tool is usually used to find compilation errors, but you can use a shell script to tell cargo-bisect-rustc what your regression is.

fadeevab commented 2 years ago

@fanninpm @apiraino

Regression in 936eba3b348e65b658b60218cc9237f02abdbeb4

==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2022-05-16 to nightly-2022-05-20
regressed nightly: nightly-2022-05-19
searched commit range: https://github.com/rust-lang/rust/compare/4c5f6e6277b89e47d73a192078697f7a5f3dc0ac...cd282d7f75da9080fda0f1740a729516e7fbec68
regressed commit: https://github.com/rust-lang/rust/commit/936eba3b348e65b658b60218cc9237f02abdbeb4

<details>
<summary>bisected with <a href='https://github.com/rust-lang/cargo-bisect-rustc'>cargo-bisect-rustc</a> v0.6.4</summary>

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc ./script.sh --start 2022-05-16 --end 2022-05-20 --preserve 

</details>

I actually bisected between 2021-11-11 and 2022-08-09.

fanninpm commented 2 years ago

regressed commit: https://github.com/rust-lang/rust/commit/936eba3b348e65b658b60218cc9237f02abdbeb4

Thank you! That points to #96867.

fadeevab commented 2 years ago

Examples of regression.

It checks random lines in the documentation comment: image

It shows 3-5% of coverage on a little error.rs file (100% without regression). image

fadeevab commented 2 years ago

@fanninpm By the way, you could probably change labels on this issue :)

fanninpm commented 2 years ago

By the way, you could probably change labels on this issue

Can I?

@rustbot label -E-needs-bisection

Nevertheless, it would be helpful to have a Minimal, Complete, and Verifiable Example.

fadeevab commented 2 years ago

@fanninpm

Steps:

  1. Download https://github.com/fadeevab/cocoon/

  2. Put script.sh into the project directory.

    #!/bin/sh

export CARGO_INCREMENTAL=0
export RUSTDOCFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"

cargo test --all-features
mkdir -p ./${CARGO_TARGET_DIR}/coverage
./grcov ./${CARGO_TARGET_DIR}/${CARGO_BUILD_TARGET}/debug/ -s . -t html --llvm --branch --ignore-not-existing --ignore "/*"  --excl-line '#\[|=> panic!|unreachable!|Io\(std::io::Error\)'  --excl-br-line '#\[|=> panic!|unreachable!|assert_..!' -o ./${CARGO_TARGET_DIR}/coverage/lcov.html
grep -e ">[89]" ./${CARGO_TARGET_DIR}/coverage/lcov.html/index.html
  3. Run bisection 
    cargo-bisect-rustc --script ./script.sh --start 2022-05-16 --end 2022-05-20 --preserve

Explanation: if grep finds 80-90% in a coverage report (it looks for a 99% or say 84%, which is also good), so it's just looking for '>9' or '>8', then the script returns 0, else it returns error 1. I double checked the coverage reports: it really finds a regression between the commits.

fadeevab commented 2 years ago

@fanninpm

A Minimal, Complete, and Verifiable Example: https://github.com/fadeevab/rust-lang-rust-issues-100125

fadeevab commented 2 years ago

@fanninpm Could "regression-untriaged" be removed?

fanninpm commented 2 years ago

@fanninpm Could "regression-untriaged" be removed?

Not until it has been triaged by the appropriate team. @apiraino may be more privy to the triaging processes that go on around here.

apiraino commented 2 years ago

I've tried spending a bit of time on the differences between grcov reports, I'm unsure how to provide a clear actionable to act on for this behaviour. I checked the grcov issue tracker and found https://github.com/mozilla/grcov/issues/725 which seems close to what is reported here. That issue lead to https://github.com/rust-lang/rust/issues/91661 which has more insights and suggestions to help forming a diagnose.

For this reason I'd leave this issue marked as in need of some more accurate triaging than I can provide.

federicomenaquintero commented 2 years ago

I can reproduce this with librsvg. Both with grcov 0.8.11:

RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Clink-dead-code -Coverflow-checks=off"

(I'm using -Zprofile because -Cinstrument-coverage produces incorrect results when linking to C code)

jieyouxu commented 1 week ago

Triage: closing because -Zprofile is removed following #131829.

fadeevab commented 5 days ago

No body, no crime 😁

For the reference, here is what can be used instead of -Zprofile: https://github.com/rust-lang/rust/issues/42524#issuecomment-2466459610