search

taiki-e / cargo-llvm-cov

Cargo subcommand to easily use LLVM source-based code coverage (-C instrument-coverage).
Apache License 2.0
933 stars 57 forks source link

Difference between `cargo test` and `cargo llvm-cov` results when using `thread_local` #208

Closed Maximkaaa closed 2 years ago

Maximkaaa commented 2 years ago

It seems to me that llvm-cov runs all the tests in a single thread. I guess there might be a good reason for that, but in some cases it can result in some tests failing which pass with cargo test. For example, if the test or the code itself uses thread_local storage for something, its contents can be different in these two cases.

Even when the cargo test command is run without parallelism (with --jobs 1), it still creates a new thread for each test. It would be nice if llvm-cov would do the same to make the test results be the same.

taiki-e commented 2 years ago

I would love to change the default behavior here once https://github.com/rust-lang/rust/issues/91092 is fixed.

Maximkaaa commented 2 years ago

Oh, I didn't even know about the -- --test-threads=1 option. Yes, this explain the difference, since -j and --test-threads work differently.

Maybe consider adding a note about this in the readme until that issue is fixed. Because it took me quite a while to figure out why my tests were passing with cargo test but not with cargo llvm-cov.

taiki-e commented 2 years ago

Maybe consider adding a note about this in the readme until that issue is fixed. Because it took me quite a while to figure out why my tests were passing with cargo test but not with cargo llvm-cov.

Sounds good to me!

taiki-e commented 2 years ago

This behavior has been documented by #209 (thanks @Maximkaaa), so I'm going to close this in favor of the upstream issue.

taiki-e commented 1 year ago

FYI, since cargo-llvm-cov 0.5.20, we no longer set RUST_TEST_THREADS and now adopt a more efficient alternative to workaround rustc bug.