It would be really nice to compute the code coverage in our CI pipeline, especially since we can upload the result to CircleCI's artifacts directory (BTW, we should propably do the same for our benchmark).
That being said, before getting there we have several changes to make if we want to do it.
Add coverage tooling (easy)
We should install a recent lcov in our Docker image for testing.
Note that it must be a recent lcov because older version don't work with GCC 8.
Make the test suite faster
In order to have a comprehensive coverage, we should run the test suite 3 times:
once without SIMD
once with SSE
once with AVX
Otherwise, some code path won't be covered by the test suite.
In order to keep a bearable user experience, we should remove the stuff that unecessarily slow don our test suite.
First, we should get rid of test_ec.sh:
it's way too slow (you can't run the test suite in Debug mode, and I'm not sure Release mode give an accurate coverage)
it's too coarse grained
test_ec.sh is based on ec_test, which could be good for integration test but not for unit testing.
I think we should convert as much as possible the test from test_ec.sh into unit tests, and for the few thing that do require a full run then we can add some integration test.
Next, we should remove most of the loop that run the tests thousands of time with random input:
that make the tests non-reproducible
that's not the role of unit test: if we want to test our behavior with random input to detect numeric instabilities or other bugs this should be done by using instrumentation-guided fuzzing (such as AFL), not by randomly adding randomness hither and yon.
Finally, we should move the benchmark into it's own CircleCI job and update the testing job to run the testing suite thrice before collecting the code coverage results.
It would be really nice to compute the code coverage in our CI pipeline, especially since we can upload the result to CircleCI's artifacts directory (BTW, we should propably do the same for our benchmark).
That being said, before getting there we have several changes to make if we want to do it.
Add coverage tooling (easy)
We should install a recent
lcovin our Docker image for testing. Note that it must be a recentlcovbecause older version don't work with GCC 8.Make the test suite faster
In order to have a comprehensive coverage, we should run the test suite 3 times:
Otherwise, some code path won't be covered by the test suite.
In order to keep a bearable user experience, we should remove the stuff that unecessarily slow don our test suite.
First, we should get rid of
test_ec.sh:test_ec.shis based onec_test, which could be good for integration test but not for unit testing.I think we should convert as much as possible the test from
test_ec.shinto unit tests, and for the few thing that do require a full run then we can add some integration test.Next, we should remove most of the loop that run the tests thousands of time with random input:
Finally, we should move the benchmark into it's own CircleCI job and update the testing job to run the testing suite thrice before collecting the code coverage results.