search

taiki-e / cargo-llvm-cov

Cargo subcommand to easily use LLVM source-based code coverage (-C instrument-coverage).
Apache License 2.0
949 stars 58 forks source link

List the tests that exercise a function #357

Open smoelius opened 8 months ago

smoelius commented 8 months ago

When someone fixes a piece of code, it is considered good practice to write a test that exercises the fix.

In many cases, an existing test could be modified to save having to write a completely new test from scratch.

How difficult do you think it would be for a tool like cargo-llvm-cov to list the tests that exercise a function?

taiki-e commented 8 months ago

I don't know a good way to do this.

I guess the simplest, stupidest, and accurate way would be to run all the tests individually and analyze the resulting JSON reports to see if they affect the coverage of the target function, but that would take too long.

If listing is not your end goal, you might be able to use the feature used in coverage guided fuzzing, but I don't know how that works.

smoelius commented 8 months ago

If listing is not your end goal, you might be able to use the feature used in coverage guided fuzzing, but I don't know how that works.

What are you referring to?

taiki-e commented 8 months ago

What are you referring to?

I was thinking about SanitizerCoverage used in LibFuzzer. However, this may not be a match for the use case here, as this seem more suited to changing the input and getting it's impact on the coverage.

https://www.llvm.org/docs/LibFuzzer.html https://clang.llvm.org/docs/SanitizerCoverage.html

smoelius commented 8 months ago

Thanks very much for your responses.

I am sensing that having the ability to list the tests that exercise a function doesn't sound valuable to you?

taiki-e commented 8 months ago

I am sensing that having the ability to list the tests that exercise a function doesn't sound valuable to you?

To be honest, it depends on what the implementation is. If it is relatively easy to implement as an extension of the current cargo-llvm-cov, then I think it is reasonable to implement it as part of the cargo-llvm-cov. Otherwise, it would make more sense to implement it as an individual tool that uses the output of the cargo-llvm-cov, like the llvm-cov-pretty.