fff needs code coverage

[wulfgarpro]

With the ever increasing PRs and additional features, fff could do with some code coverage stats.

This is a perfect write up on gcov for TravisCI and OpenCppCoverage for AppVeyor.

https://codingnest.com/how-to-get-code-coverage-from-ci/

[meekrosoft]

Sounds like a great idea....although I wonder how good the coverage data will be? If we don't use a pre-processor macro it's code definition will not end up in a translation unit. So maybe we need to force instantiate the macros somehow?

[wulfgarpro]

Isn't that what the tests do?

On Thu., 6 Dec. 2018, 4:31 pm Mike Long <notifications@github.com wrote:

Sounds like a great idea....although I wonder how good the coverage data will be? If we don't use a pre-processor macro it's code definition will not end up in a translation unit. So maybe we need to force instantiate the macros somehow?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/meekrosoft/fff/issues/68#issuecomment-444753448, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6TtGYKqiHgcRwLsnX7awJcUK6T5Hk9ks5u2Ku2gaJpZM4ZFJBY .

[meekrosoft]

Sorry, I'm not being very clear :-D What I mean is, there isn't really any executable code in fff...it's just a header file that contains macros which may instantiate data structures. So I'm struggling to see what code coverage can help with in this case?

[wulfgarpro]

Yes, you're correct. The preprocessor gets in the way of coverage testing. I'll need to rethink this.

[meekrosoft]

Might still be useful to ensure the tests actually get run though?

[wulfgarpro]

@meekrosoft, I had a little look at this just now, a cursory idea; haven't thought hard enough about reasons this might not be useful:

gcc -E fff.h > out1.txt
gcc -E test/fff_test_c.c > out2.txt
awk 'a[$0]++' out1.txt out2.txt | wc -l  # 2834
cat out1.txt | wc -l  # 327
cat out2.txt | wc -l  # 3898
# total lines: 327+3898=4225
# percentage of match: 2834/4225*100=~67%

With this method, we could calculate an average across all test files and use it as a heuristic.

[wulfgarpro]

NOTE: the above method can't work given that the pre-processed c file, e.g. test/fff_test_c.c includes all of fff.h.

Another approach that I'm playing with (haven't a good result yet):

#!/usr/bin/env bash

fff_len=`cat fff.h | wc -l`

# test/fff_test_c.c
gcc -E test/fff_test_c.c > out
comm -13 <(sort < fff.h | uniq) <(sort < out | uniq) > unique_to_out      # strip out fff.h include
awk 'FNR==NR{a[$1];next}($1 in a){print}'  unique_to_out fff.h > matched  # find unique matches
grep -Fxf  <(sort < matched | uniq) <(sort < fff.h | uniq) > coverage     # find coverage of matches in fff.h
cov_len=`cat coverage | wc -l`
echo "test/fff_test_c.c coverage:"
echo "$cov_len / $fff_len" | bc  -l

# test/fff_test_cpp.cpp
gcc -E test/fff_test_cpp.cpp -I . > out
comm -13 <(sort < fff.h | uniq) <(sort < out | uniq) > unique_to_out
awk 'FNR==NR{a[$1];next}($1 in a){print}'  unique_to_out fff.h > matched
grep -Fxf  <(sort < matched | uniq) <(sort < fff.h | uniq) > coverage
cov_len_2=`cat coverage | wc -l`
echo "test/fff_test_cpp.cpp coverage:"
echo "$cov_len_2 / $fff_len" | bc  -l

# total coverage
echo "TOTAL COVERAGE:"
echo "($cov_len + $cov_len_2) / ($fff_len * 2)" | bc -l

[meekrosoft]

Mmm, interesting approach....