Open andrewrk opened 7 years ago
Based on my unpleasant experience with gcov (GCC coverage tool) in foolish drive for 100% coverage: there should be some easy way to label certain lines or whole functions as "doesn't need coverage". All kinds of unlikely situations, impossible to emulate.
gcov doesn't support anything like this, so I write a tool which scanned all C source files, searched for "no coverage" tags and removed these lines from gcov output.
It looked like:
if (...) {
exit(1); //-- (comment "//--" means no coverage for this line)
}
// The whole function is taken out of coverage via the //-->> and //<<-- brackets
void foo(void) { //-->>
...
} //<<--
There should be also some way to skip coverage checking for 3pp libraries, w/o touching their source code.
Just though I'd mention that kcov works perfectly as is with zig at the moment. I think coverage in general is slightly more useful than it normally would be for zig as well, since it clearly shows what segments of code were never compiled. This helps with compile-time evaluation heavy bits.
Only steps required are:
zig test bigint.zig
kcov kcov-out ./zig-cache/test
firefox kcov-out/index.html
Oh that's really interesting with the defers in yellow. The number tells you how many different ways there were to exit the block, and it only goes green if you manage to hit them all (in this case by having each try
activate the error handling). Same thing with try
, it'll be yellow unless you managed to test the error return.
The defer thing might change with #283
I tried this using zig test
on the std lib. It's really quite useful, but I see some confusing output. In my test results I see:
Test 341/466 os.path.join...OK
Test 342/466 os.path.isAbsoluteWindows...OK
Test 343/466 os.path.isAbsolutePosix...OK
Test 344/466 os.path.windowsParsePath...OK
Test 345/466 os.path.resolve...OK
Test 346/466 os.path.resolveWindows...OK
Test 347/466 os.path.resolvePosix...OK
Test 348/466 os.path.dirnamePosix...OK
Test 349/466 os.path.dirnameWindows...OK
Test 350/466 os.path.basename...OK
Test 351/466 os.path.relative...OK
Test 352/466 os.path.real...OK
but in the coverage report I see no coverage:
I wonder what's going on here.
Looking forward to this option 🚀
kcov still works, the instructions have changed slightly as ./zig-cache/test
doesn't exist.
rm -rf zig-cache/o
zig test lib/std/bit_set.zig
kcov kcov-out ./zig-cache/o/*/test
open kcov-out/index.html
It is very slow on my M1 Macbook Air. Took over 5 minutes.
It would be interesting to generate a coverage report based on the kcov
one* where the different logical possibilities and ramifications are informed in detail, as Jest does:
(informing the coverage of "statements", "branch", "functions", "lines" and their logical ramifications)
-----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files | 100 | 100 | 100 | 100 |
index.js | 100 | 100 | 100 | 100 |
server.js | 100 | 100 | 100 | 100 |
-----------|---------|----------|---------|---------|-------------------
Test Suites: 2 passed, 2 total
Tests: 11 passed, 11 total
Snapshots: 0 total
Time: 3.735 s, estimated 5 s
Ran all test suites.
*(kcov seems to provide everything that's necessary for this)
I tried this using
zig test
on the std lib. It's really quite useful, but I see some confusing output. In my test results I see:Test 341/466 os.path.join...OK Test 342/466 os.path.isAbsoluteWindows...OK Test 343/466 os.path.isAbsolutePosix...OK Test 344/466 os.path.windowsParsePath...OK Test 345/466 os.path.resolve...OK Test 346/466 os.path.resolveWindows...OK Test 347/466 os.path.resolvePosix...OK Test 348/466 os.path.dirnamePosix...OK Test 349/466 os.path.dirnameWindows...OK Test 350/466 os.path.basename...OK Test 351/466 os.path.relative...OK Test 352/466 os.path.real...OK
but in the coverage report I see no coverage:
I wonder what's going on here.
kcov
uses DWARF debugging information for compiled programs to make it possible to collect coverage information without special compiler switches.
In https://www.youtube.com/watch?v=1QMHbp5LUKg Simon Kagstrom says that he found that debugging information is often broken on Linux systems, because kcov
uses debug information that are not usually used by debuggers (and thus not well tested).
Add a
--coverage
argument tozig test
and have it generate a nice report.Related:
20989
20992