Files/modules behind feature flags are not covered by integration tests

[dnaka91]

Describe the bug

When a module is behind a feature flag and tested by an integration test (separate file in the tests folder), the coverage data for that module is missing. Other modules that are not behind a feature flag and tested by the same integration test have coverage data as expected.

This works for unit tests that are directly within the file in a tests submodule though.

I stumbled upon this in one of my projects when I put a file/module behind a feature flag and noticed that the file never came up in the coverage report even though it's used in the tests (https://github.com/dnaka91/obws).

Maybe this has something to do with cargo compiling and running integration tests as individual binaries instead of a single unit test binary that's run once.

To Reproduce

Create a new library project with two trivial functions, one in a submodule that is gated behind a feature flag. Then add a separate integration test for each function.

Structure

├── Cargo.toml
├── src
│  └── lib.rs
└── tests
   ├── add.rs
   └── mul.rs

Cargo.toml

[package]
name = "tarpaulin-test"
version = "0.1.0"
edition = "2018"

[features]
mul = []

[[test]]
name = "mul"
required-features = ["mul"]

src/lib.rs

pub fn add(a: u32, b: u32) -> u32 {
    a + b
}

#[cfg(feature = "mul")]
pub mod mul {
    pub fn mul(a: u32, b: u32) -> u32 {
        a * b
    }
}

tests/add.rs

fn main() {
    assert_eq!(5, tarpaulin_test::add(2, 3));
}

tests/mul.rs

fn main() {
    assert_eq!(6, tarpaulin_test::mul::mul(2, 3));
}

Finally, execute Tarpaulin to generate a coverage report, enabling all features:

cargo tarpaulin --out Html --out Lcov --all-features

Although all test ran, coverage for the whole mul module is missing and reported as 0%.

Expected behavior The coverage data for the module behind a feature flag should be available even if tested through separate integration tests.

[xd009642]

So if I add #[test] as an attribute to your main functions I get 100% coverage with tarpaulin and the tests run. If I don't add the attribute then the tests don't run in cargo test and they don't run with tarpaulin - which I'd expect. Also, this is the first time I've seen a main in an integration test as if it was a binary

Output from after I added the #[test] attribute about the main functions:

INFO cargo_tarpaulin::report: Coverage Results:
|| Tested/Total Lines:
|| src/lib.rs: 4/4 +100%
|| tests/add.rs: 2/2 +100%
|| tests/mul.rs: 2/2 +100%
||
100.00% coverage, 8/8 lines covered, +100% change in coverage

If you're seeing anything different or your project is doing something different if you let me know the specific file that disappeared with the feature I can have a look at that specifically.

[dnaka91]

Sorry, it seems I made a mistake in my sample and forgot to add the #[test] attributes. In my project I use #[tokio::test] (https://github.com/dnaka91/obws/blob/main/tests/general.rs) for the coverage but that shouldn't make much of a difference right?

I tested whether renaming from main to something else would maybe make a difference as that name has a special meaning to cargo, but still no difference.

I always get this output:

|| Tested/Total Lines:
|| src/client/general.rs: 27/51 +0%
|| src/client/media_control.rs: 33/50 +0%
|| src/client/mod.rs: 104/136 +0.17429193899781703%
|| src/client/outputs.rs: 14/18 +0%
|| src/client/profiles.rs: 11/14 +0%
|| src/client/recording.rs: 17/25 +0%
|| src/client/replay_buffer.rs: 0/21 +0%
|| src/client/scene_collections.rs: 11/14 +0%
|| src/client/scene_items.rs: 18/33 +0%
|| src/client/scenes.rs: 19/28 +0%
|| src/client/sources.rs: 82/152 +0%
|| src/client/streaming.rs: 0/30 +0%
|| src/client/studio_mode.rs: 21/27 +0%
|| src/client/transitions.rs: 16/46 +0%
|| src/common.rs: 4/10 +0%
|| src/de.rs: 102/112 +0%
|| src/requests/mod.rs: 30/37 +0%
|| src/requests/ser.rs: 41/43 +0%
|| src/responses/de.rs: 36/42 +0%
|| tests/common/mod.rs: 76/76 +0%
|| tests/general.rs: 20/20 +0%
|| tests/media_control.rs: 24/24 +0%
|| tests/outputs.rs: 9/9 +0%
|| tests/profiles.rs: 10/10 +0%
|| tests/recording.rs: 20/20 +0%
|| tests/scene_collections.rs: 14/14 +0%
|| tests/scene_items.rs: 49/49 +0%
|| tests/scenes.rs: 21/21 +0%
|| tests/sources.rs: 86/86 +0%
|| tests/studio_mode.rs: 17/17 +0%
|| tests/transitions.rs: 12/12 +0%
|| 
75.70% coverage, 944/1247 lines covered, +0.019501056146310702% change in coverage

And I would expect to see a src/events.rs as well, which is the missing file in the coverage. Now asking you to run this unit tests is probably asked too much as my project has quite an extensive setup to get the tests working (due to OBS).

[xd009642]

I see what it is, src/events.rs doesn't have any coverable code it's all type definitions. Given there's no functions or methods there's no lines in that file that can be covered

[dnaka91]

Oh that's what it means 🤦. I'm pretty new to using tarpaulin and thought coverage is available for enums as well. That explains why the HTML version states 0 / 0 as well and doesn't show a coverage percentage.

Sorry to steal your time for such an obvious answer.