xd009642 / tarpaulin

A code coverage tool for Rust projects
https://crates.io/crates/cargo-tarpaulin
Apache License 2.0
2.5k stars 180 forks source link

Having 1 line program produces incorrect analysis? #988

Open zhiburt opened 2 years ago

zhiburt commented 2 years ago

Hi @xd009642 Thank you for tarpaulin.

I was trying to get into DWARF a little bit (not successfully :disappointed: yet) And I was experimenting with ddbug a bit (also uses gimli). And I noticed that it doesn't captures columns in function definitions. Meaning DW_AT_decl_column is not generated. I did verified it via objdump as well.

And then I started to question myself well how tarpaulin does a match between source code and binary code.

From what I've been able to understand its done only via lines and path correct?

The implications of all of these is the following.

Describe the bug

When cargo fmt isn't applied there's a chance that tarpaulin may not cover all functions?

fn SomeFunction() { println!("123123"); }

fn SomeFunction1() { println!("123123"); }

#[test] fn test_SomeFunction() { SomeFunction(); } 
|| Tested/Total Lines:
|| src/lib.rs: 2/3 +0.00%
||
66.67% coverage, 2/3 lines covered, +0% change in coverage
fn SomeFunction() { println!("123123"); } fn SomeFunction1() { println!("123123"); } #[test] fn test_SomeFunction() { SomeFunction(); } 
|| Tested/Total Lines:
|| src/lib.rs: 1/1 +0.00%
||
100.00% coverage, 1/1 lines covered, +0% change in coverage

I would expect that both cases have the same coverage. (I might be wrong here though)


The issue I am trying to address is not actually about the coverage.

But it seems like there's a chance that tarpaulin might is not able to locate a correct function which is called in case there's more then 2 functions on the same line because of the issue I described above.

Not sure it might be a duplicate of some kind.

Thanks one more time. Take care.

xd009642 commented 2 years ago

So currently tarpaulin does "line coverage" where it is mapped based on source lines. There are other metrics like function, branch, condition, statement or MC/DC coverage which break up the code into different units where formatting wouldn't change the results. Unfortunately, tarpaulin doesn't support any of them yet, but they are on the roadmap (branch, condition and MC/DC have been a goal for like 5 years).

If you figure out a way to map the debug symbols to something more fine grained than lines I'd love to see it though and start to figure out how we can apply it to tarpaulin :smile:

zhiburt commented 2 years ago

If you figure out a way to map the debug symbols to something more fine grained than lines I'd love to see it though and start to figure out how we can apply it to tarpaulin smile

As I said I was thinking that column + line could be used. It seems like conditions, branches can be covered by it as well. (in terms of mapping) But as I said column debug symbols is not generated for some reason.

Do you think this issue can be closed or you'd like to keep it open?

PS: But isn't branches, statements etc. would involve an attachment to a tests process no? or the question is more about memory addressing in such case?

zhiburt commented 2 years ago

It seems like it must be generated? Not an expert in rustc and corresponding stuff.

https://github.com/rust-lang/rust/blob/4767ccec935824fa5d08ce3502b233d7a66adec1/compiler/rustc_codegen_cranelift/src/debuginfo/line_info.rs#L113