search

xd009642 / tarpaulin

A code coverage tool for Rust projects
https://crates.io/crates/cargo-tarpaulin
Apache License 2.0
2.5k stars 180 forks source link

tarpaulin fails to run doc-test in readme #924

Open danieleades opened 2 years ago

danieleades commented 2 years ago

I have a doc-test in my README, which i'm including like so-

// lib.rs
#![doc = include_str!("../README.md")]

this works a treat for making sure my readme stays aligned with the code, but tarpaulin doesn't seem to like it.

You can see the job output here- https://github.com/danieleades/config2/runs/4693934507?check_suite_focus=true

the code example has a no_run annotation, which cargo test is respecting, but tarpaulin apparently isn't

danieleades commented 2 years ago

i'm sorry to keep popping up like this!

danieleades commented 2 years ago

tried adding some test cases in #926, as i thought it was time you got a pull request from me instead of just issues. It's not a complete test though.

From the code I would imagine it should work, since you're just searching an entire file for the text 'no_run'. Shouldn't matter how the attributes are formatted.

Which i guess means the actual issue is a little more subtle

xd009642 commented 2 years ago

Ah I gave it a moments thought and I see what happened. So when you run cargo test for doc tests the no-run etc is chosen after it builds the doc tests. But I don't get that meta information. And the binary name for the generated doc test is generated from file name and line number of the doc test (with an addition number at the end as name collisions are possible). So I get a list of doc tests and from them work out potential source locations for the doc test. And then map the attributes at those locations to the guessed source location (guessed because there can be collisions).

Now obviously this doesn't currently handle include_str! - or indeed any include statement at all. Meaning those line counts will be wrong and no_run attributes can't be mapped back to the correct locations. The simple solution is probably to do a pass over the loaded files and spot any includes and do the includes manually on the file contents to create a post-processed file as I don't currently see a way to sort out that information using the tools cargo gives us for doc tests. It shouldn't be too difficult just mildly faffy.

Also, for generated code that may be in include!(concat!(env!("RUST_OUT"), "/generated.rs")); from libraries like prost any doc tests after the include statement will likely have an incorrect attribute mapping... But I think that's even harder to solve and may just be relegated to a TODO comment for now for the next time someone encounters this issue.

I won't have time today or tomorrow to work on this, but I'll aim to get to it during the week at some point. If you want to work on it though I'm fine with that just ideally let me know so there's not too much duplicated effort :sweat_smile:

danieleades commented 2 years ago

Any possibility of better upstream support from Cargo?

xd009642 commented 2 years ago

Potentially, I believe the code responsible for it is in rustc, it is an area which currently has no stability guarantees (to my knowledge). But I'll raise something this week and see if anyone has objections