Feature Request: Measure coverage of higher level tests

[KantarBruceAdams]

I have tests at multiple levels. unit tests in _test.go manner program tests which run a copy of the program itself

I place these in suite in program_test.go and they are run via go test as normal though this is not the only way.

In the unit tests a lot of important functionality is mocked leading to low apparent test coverage. The added coverage from program tests is not included in the output of go test -c or gobco.

This could be supported if an instrumented version of the program could be generated so that program level integration tests generate statistics that can be included in the final coverage report. To do this we would need to:

1. generate an instrumented version of the program (similar to -ftest-coverage for gcov for C/c++)
2. record the results somewhere (e.g. to a file or via a socket)
3. gobco adds the results as for any other kind of test

The instrumented files created by gobco probably solve the hardest part of this already. Helpers are needed that:

• write statistics,
• read and accumulate statistics
• report statement and branch coverage (as gobco does now) based on stored statistics

[rillig]

Sounds reasonable, and if you have looked at the code, you may have noticed the TODO comments for generating JSON output instead of human-readable text.

You can run gobco -- -keep to prevent the generated files in the temporary directory from being deleted. There, the file gobco.go contains the slice of conditions that are tested, just in case you want to have a look at how gobco works internally.

The part I'm not entirely sure about is: when exactly should the statistics be written? There may be programs that call os.Exit, and I don't know how to intercept this call. For all the other programs, it may be enough to insert a defer gobcoWriteStatistics() at the beginning of the main function.

Of course, that function is not yet written, but given the current restructured code, it shouldn't be too difficult. I'm having a look at it.

[rillig]

@KantarBruceAdams Do you have an example project for me to play with? I think I understand what you want to achieve, but to be extra sure it's better to have an example project.

I think gobco is ready for the final cut, all the ingredients should be implemented by now.

[rillig]

Feedback timeout.

[KantarBruceAdams]

Sorry for the late reply. I do not have an example project I can share at the moment. I may be able to create one at a later date.