search

SimonKagstrom / kcov

Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options
http://simonkagstrom.github.io/kcov/
GNU General Public License v2.0
712 stars 109 forks source link

Code coverage from an unexecuted code #318

Closed oleksandrkozlov closed 4 years ago

oleksandrkozlov commented 4 years ago

During calculating the test coverage percentage, kcov does not take into account files from which code was not called in the tests. But it takes into account the code from a file if at least one function was called from it. So if we have a dozen source files and write a test for all functions only from one file, we will have 100% code coverage, which is not true. In order to have a correct code coverage, we need to write tests at least for one function per each file.

My question would be: is this behavior intentional?

SimonKagstrom commented 4 years ago

Are we talking about compiled code here? If so, this is a linker issue and not something kcov does by itself. The linker will typically remove any non-referenced portions of the code, so I think this is what happens here.

For bash and Python it's a different situation. For bash, kcov will try to find other source files in the same directory as the main script, but for Python, the situation you describe might well occur.

oleksandrkozlov commented 4 years ago

Yes, I was talking about compiled code: C/C++.

oleksandrkozlov commented 4 years ago

So if I understand you correctly this behavior was not intentional. The problem is with an approach that kcov uses during the test coverage calculation. It relies on the linker. So if we test a library that has many functions divided by many files, and we don't call all those functions in our test binary, during the linkage stage the linker will remove all unused symbols from that binary and we will not be able to know that that uncovered code existed.

SimonKagstrom commented 4 years ago

Yes, it sounds like it's how the binary is built. kcov doesn't know anything about what sources are included beforehand - it simply parses the binary to gather all sources for which there are compiled code.

If there are unused functions, the compiler and linker can cleverly remove them to save space. In that case, kcov also doesn't see them. You can sometimes see this in covered files as well, where portions of the code is greyed out, simply because it's "dead".

This is sometimes an effect of optimizations, so looking at the build options might help.

oleksandrkozlov commented 4 years ago

Thank you for the clarification.

davidtazy commented 3 years ago

Hello, I do not understand why the fix is not possible: Command line give the list of directory to "cover", kcov can detect which files are covered. So it should be possible to deduce all the file not covered and show them?

SimonKagstrom commented 3 years ago

Well, the source files for the program are actually given by the debug info in the binary, so that's what kcov looks at. Non-compiled source files are simply not parsed at all.

Of course, it would be possible to scan some directories and look for other source files (for bash, it does this), but that would give a lot of false negatives since the source files are probably not meant to be included in this particular program.