xd009642 / tarpaulin

A code coverage tool for Rust projects
https://crates.io/crates/cargo-tarpaulin
Apache License 2.0
2.5k stars 180 forks source link

Run type Doctests missing CARGO_MANIFEST_PATH #761

Closed pinkforest closed 3 years ago

pinkforest commented 3 years ago

Description

When running --run-types Doctests the CARGO_MANIFEST_PATH is missing

Trying to mitigate it with --manifest-path also evaluates to env CARGO_MANIFEST_PATH missing.

On run type Test env CARGO_MANIFEST_PATH is visible just like running the cargo test

I initially ran to this issue here: https://github.com/alexliesenfeld/httpmock/issues/45

cargo version 1.52.0 and 1.54.0-nightly (070e459c2 2021-05-11) same result

Reproduce

I created an environment to reproduce here which has both run type Tests and Doctests: https://github.com/pinkforest/test_tarpaulin_env

Code under test: src/lib.rs has both Doctest and Tests Run types

// doctest
/// Get my own package path for e.g. setting configuration from
/// ```rust
/// # use test_tarpaulin_env::*;
/// #
/// # fn main() {
///     let test_path = get_my_path("test_data");
///     println!("test_path evaluated to = lossy?:{}", test_path.display());
/// # }
/// ```
// stole below from tarpaulin::tests::utils.rs to illustrate
// tarpaulin has test workspace.rs that calls it but there env CARGO_MANIFEST_DIR evaluates correctly
pub fn get_my_path(test_dir_name: &str) -> PathBuf {
    let mut test_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap());
    test_dir.push("tests");
    test_dir.push("data");
    test_dir.push(test_dir_name);
    test_dir
}

#[cfg(test)]
mod tests {
  use super::*;

  #[test]
  fn cargo_manifest_path() {
    let test_dir = get_my_path("");
    println!("Test dir evaluated to = {}", test_dir.display());
    assert_ne!(test_dir, PathBuf::from(""));
  }
}

Run type: Doctests

env cargo tarpaulin --run-types Doctests -v

Doctests will fail with missing CARGO_MANIFEST_PATH due to panic!() from get_my_path I stole from your tests/util.rs

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo tarpaulin --run-types Doctests -v
May 17 09:10:46.988 DEBUG cargo_tarpaulin: set up logging
May 17 09:10:46.988  INFO cargo_tarpaulin::config: Creating config
May 17 09:10:46.993  INFO cargo_tarpaulin: Running Tarpaulin
May 17 09:10:46.993  INFO cargo_tarpaulin: Building project
   Compiling test_tarpaulin_env v0.1.0 (/home/foobar/testing-rust/test_tarpaulin_env)
    Finished test [unoptimized + debuginfo] target(s) in 0.28s
   Doc-tests test_tarpaulin_env
May 17 09:10:47.496  INFO cargo_tarpaulin: Launching test
May 17 09:10:47.496  INFO cargo_tarpaulin: running /home/foobar/testing-rust/test_tarpaulin_env/target/doctests/src_lib_rs_17_0/rust_out
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotPresent', src/lib.rs:29:69
stack backtrace:
   0: rust_begin_unwind
             at /rustc/8cf990c9b5c59f25c806fad9f4466f9d6509bbea/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/8cf990c9b5c59f25c806fad9f4466f9d6509bbea/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/8cf990c9b5c59f25c806fad9f4466f9d6509bbea/library/core/src/result.rs:1355:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/8cf990c9b5c59f25c806fad9f4466f9d6509bbea/library/core/src/result.rs:1037:23
   4: test_tarpaulin_env::get_my_path
             at ./src/lib.rs:29:38
   5: rust_out::main
             at ./src/lib.rs:7:20
   6: core::ops::function::FnOnce::call_once
             at /rustc/8cf990c9b5c59f25c806fad9f4466f9d6509bbea/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Error: "Test failed during run"

Even if I try --manifest-path=. it fails with relative ./Cargo.toml:

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo tarpaulin --manifest-path=./Cargo.toml --run-types Doctests -v
--snip--
ay 17 09:18:48.620  INFO cargo_tarpaulin: running /home/foobar/testing-rust/test_tarpaulin_env/target/doctests/src_lib_rs_17_0/rust_out
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotPresent', src/lib.rs:29:69
--snip--

Or even absolute --manifest-path fails:

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo tarpaulin --manifest-path=/home/foobar/testing-
rust/test_tarpaulin_env/Cargo.toml --run-types Doctests -v
May 17 09:20:08.964  INFO cargo_tarpaulin: running /home/foobar/testing-rust/test_tarpaulin_env/target/doctests/src_lib_rs_17_0/rust_out
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotPresent', src/lib.rs:29:69

Tarpaulin did think itself previously the manifest was correctly set as tarpaulin does some checking:

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo tarpaulin --manifest-path=/error_error --run-types Doctests -v
May 17 09:21:23.381  WARN cargo_tarpaulin::config: Couldn't get project metadata `cargo metadata` exited with an error: error: the manifest-path must be a path to a Cargo.toml file

Expectation

On both Doctests and Tests expectation is env CARGO_MANIFEST_PATH evaluates correctly.

Considering:

Doc README.md was initially suggested via PR to make expectation clear not to rely on env CARGO_MANIFEST_PATH:

Which in turn with both code and resp clarifies CARGO_MANIFEST_PATH env should be set despite:

Regular Cargo test:

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo test

Cargo test evaluates both correctly using CARGO_MANIFEST_PATH and test succeeds:

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo test
   Compiling test_tarpaulin_env v0.1.0 (/home/foobar/testing-rust/test_tarpaulin_env)
    Finished test [unoptimized + debuginfo] target(s) in 0.18s
     Running unittests (target/debug/deps/test_tarpaulin_env-5a8f46bc4bedec67)

running 1 test
test tests::cargo_manifest_path ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests test_tarpaulin_env

running 1 test
test src/lib.rs - get_my_path (line 17) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s

Run type: Tests

env cargo tarpaulin --run-types Tests -v

Tests finds that one unit in cfg test and we see 100% coverage with tarpaulin run mode Tests

foobar@foobar-MS-7C77:~/testing-rust/test_tarpaulin_env$ env cargo tarpaulin --run-types Tests -v
May 17 09:01:30.722 DEBUG cargo_tarpaulin: set up logging
May 17 09:01:30.722  INFO cargo_tarpaulin::config: Creating config
May 17 09:01:30.880  INFO cargo_tarpaulin: Running Tarpaulin
May 17 09:01:30.880  INFO cargo_tarpaulin: Building project
    Finished test [unoptimized + debuginfo] target(s) in 0.10s
May 17 09:01:31.012  INFO cargo_tarpaulin: Launching test
May 17 09:01:31.012  INFO cargo_tarpaulin: running /home/foobar/testing-rust/test_tarpaulin_env/target/debug/deps/test_tarpaulin_env-5a8f46bc4bedec67

running 1 test
test tests::cargo_manifest_path ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

May 17 09:01:31.116  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| Tested/Total Lines:
|| src/lib.rs: 9/9 +0%
|| 
100.00% coverage, 9/9 lines covered, +0% change in coverage
pinkforest commented 3 years ago

Tested fix from: #764

And it works now! great @xd009642 fixing it.

Tested on: https://github.com/pinkforest/test_tarpaulin_env

Doctests;

env RUN_MODE=development cargo-tarpaulin tarpaulin --run-types Doctests -v
May 19 13:52:40.343 DEBUG cargo_tarpaulin: set up logging
May 19 13:52:40.540  INFO cargo_tarpaulin: Running Tarpaulin
May 19 13:52:40.540  INFO cargo_tarpaulin: Building project
May 19 13:52:40.540  INFO cargo_tarpaulin::cargo: Cleaning project
    Finished test [unoptimized + debuginfo] target(s) in 0.19s
   Doc-tests test_tarpaulin_env
May 19 13:52:41.009  INFO cargo_tarpaulin::process_handling::linux: Launching test
May 19 13:52:41.009  INFO cargo_tarpaulin::process_handling: xx running /home/foobar/testing-rust/test_tarpaulin_env/target/doctests/src_cargo_manifest_path_rs_8_0/rust_out
test_dir evaluated to = lossy?:/home/foobar/testing-rust/test_tarpaulin_env/tests/data/test_data
May 19 13:52:41.061  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| Tested/Total Lines:
|| src/cargo_manifest_path.rs: 6/6 +0%
|| 
100.00% coverage, 6/6 lines covered, +0% change in coverage

Tests:

env RUN_MODE=development cargo-tarpaulin tarpaulin --run-types Tests -v
May 19 13:52:54.191 DEBUG cargo_tarpaulin: set up logging
May 19 13:52:54.387  INFO cargo_tarpaulin: Running Tarpaulin
May 19 13:52:54.387  INFO cargo_tarpaulin: Building project
May 19 13:52:54.387  INFO cargo_tarpaulin::cargo: Cleaning project
   Compiling test_tarpaulin_env v0.1.0 (/home/foobar/testing-rust/test_tarpaulin_env)
    Finished test [unoptimized + debuginfo] target(s) in 0.29s
May 19 13:52:54.795  INFO cargo_tarpaulin::process_handling::linux: Launching test
May 19 13:52:54.795  INFO cargo_tarpaulin::process_handling: xx running /home/foobar/testing-rust/test_tarpaulin_env/target/debug/deps/test_tarpaulin_env-5a8f46bc4bedec67

running 1 test
test cargo_manifest_path::tests::cargo_manifest_path ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

May 19 13:52:54.902  INFO cargo_tarpaulin::report: Coverage Results:
|| Uncovered Lines:
|| Tested/Total Lines:
|| src/cargo_manifest_path.rs: 9/9 +0%
|| 
100.00% coverage, 9/9 lines covered, +0% change in coverage

And works with my repo gis_puller now. good stuff - there is one unit that is funky but it's unrelated to this.