eddyxu / cpp-coveralls

Upload gcov results to coveralls.io
Apache License 2.0
282 stars 74 forks source link

Build process needs to run twice to produce a working coverage report #111

Open hhirsch opened 7 years ago

hhirsch commented 7 years ago

I have fiddled with support for code coverage for our project https://github.com/GlPortal/RadixEngine/pull/30

However it only started producing proper results after I ran the build process twice https://github.com/GlPortal/RadixEngine/pull/30/commits/59a6968cc12718b0550809ac239d83041c15be15

Any idea what the problem is?

Here is a log of the build: https://travis-ci.org/GlPortal/RadixEngine/builds/212253813

chrisosaurus commented 6 years ago

Strange, from your travis log I see 2 interesting sections

The first time coveralls only analyses one files, the second time it analyses 'a lot'.

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L506

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
Lines executed:0.00% of 7
Creating 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

# it then moves on to doing other things

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L715

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

/data/external/json11/CMakeFiles/json11.dir/json11.cpp.gcda:cannot open data file, assuming not executed

File '/usr/include/c++/6.3.1/bits/stl_algobase.h'
Lines executed:0.00% of 23
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#stl_algobase.h.gcov'

File '/usr/include/c++/6.3.1/bits/predefined_ops.h'
Lines executed:0.00% of 1
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#predefined_ops.h.gcov'

[...]
# lots more files analysed

Very strange.

chrisosaurus commented 6 years ago

Attempted reproduction steps roughly following https://github.com/GlPortal/RadixEngine/blob/master/COMPILE.md (I tried to use a build dir at first but it didn't seem to add anything, so tried without)

$ sudo pacman -S cmake make gcc pkg-config assimp libepoxy sdl2{,_mixer} bullet mesa tinyxml2 freeimage
$ sudo pip install cpp-coveralls
$ git clone https://github.com/GlPortal/RadixEngine.git
$ cd RadixEngine
$ git submodule init
$ git submodule update
$ cmake .
$ make
$ make test
$ coveralls --dump HERE
$ grep -o '"name":[^,]*,' HERE | wc -l
890
$ grep -o '"name":[^,]*,' HERE | head -10
"name": "include/radix/Camera.hpp",
"name": "include/radix/SimulationManager.hpp",
"name": "include/radix/EntityManager.hpp",
"name": "include/radix/Entity.hpp",
"name": "include/radix/Console.hpp",
"name": "include/radix/SoundManager.hpp",
"name": "include/radix/BaseGame.hpp",
"name": "include/radix/World.hpp",
"name": "include/radix/GameWorld.hpp",
"name": "include/radix/Viewport.hpp",

so it seems to be correctly generating coverage report sfor 890 source files after a single run.

Are you still able to produce this? if so please provide reproduction steps.

hhirsch commented 6 years ago

The test target does not exist so make test should just fail. make should build the tests though.

iirc coveralls not running wasn't the problem but it reporting 0% coverage when there is 3% coverage.

Variations can be easily tested on a branch and I'll be happy to change the line accordingly: https://github.com/GlPortal/RadixEngine/blob/master/.travis.yml

I tried to tackle this problem multiple times and I always had to revert to the run twice method.

chrisosaurus commented 6 years ago

make test seems to run the tests for RadixEngine

[chris@niflheim RadixEngine]$ git remote -vv
origin  https://github.com/GlPortal/RadixEngine.git (fetch)
origin  https://github.com/GlPortal/RadixEngine.git (push)

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

Am I misunderstanding something?

I guess I can try find time to setup my own coveralls and travis for a clone of RadixEngine and see what I can get out of it

chrisosaurus commented 6 years ago

So in RadixEngine it looks like:

[chris@niflheim RadixEngine]$ make tests
[  3%] Built target RadixEntity
[  3%] Built target GLAD
[  4%] Built target json11
[ 11%] Built target VHACD_LIB
[ 33%] Built target angelscript
[ 36%] Built target easy_profiler
[ 88%] Built target RadixEngine
[ 93%] Built target MathTest
[ 96%] Built target XmlHelperTest
[100%] Built target QuaternionTest
[100%] Built target tests

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

[chris@niflheim RadixEngine]$ ctest
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

I've setup travis and coveralls for my clone, will play and around and then report results back, might take a while

chrisosaurus commented 6 years ago

As a side effect of investigating this I noticed an issue in that you have a public .coveralls.yml (only needed for private travis, not needed for open source.)

I created an issue on your repo to describe this https://github.com/GlPortal/RadixEngine/issues/173 and then sent a PR to fix this https://github.com/GlPortal/RadixEngine/pull/174

That should just work as-is unless I am misunderstanding something.

chrisosaurus commented 6 years ago

Okay, I now have coveralls running outside the docker container and I'm able to reproduce the 0% coveralls report on my clone.

chrisosaurus commented 6 years ago

tests and output:

Running:

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401344880/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401348185/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls --dryrun -i include -i source -e tests --gcov-options \'-lp\'
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401348813/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401349052/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Very strangely, only this last one produced a huge log. All previous versions produced

1.24s$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'
{u'url': u'https://coveralls.io/jobs/38190001', u'message': u'Job #11.1'}

only the last one (running cmake/make/ctest twice, coveralls once) produces the output we expect

[0K$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcno:version '603*', prefer '408*'
/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcda:cannot open data file, assuming not executed
File '/usr/include/c++/6.3.1/bits/stl_tree.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#stl_tree.h.gcov'

File '/usr/include/c++/6.3.1/bits/alloc_traits.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#alloc_traits.h.gcov'

[...]

File '/usr/include/c++/6.3.1/atomic'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#atomic.gcov'

File '/usr/include/c++/6.3.1/bits/atomic_base.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#atomic_base.h.gcov'

File '/data/external/easy_profiler/easy_profiler_core/include/easy/profiler.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###data#external#easy_profiler#easy_profiler_core#include#easy#profiler.h.gcov'

File '/usr/include/c++/6.3.1/bits/locale_facets.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#locale_facets.h.gcov'

{u'url': u'https://coveralls.io/jobs/38190033', u'message': u'Job #13.1'}

but it still seemed to generate a coverage of 0% unless something else went wrong.

I think the problem is at least in part due to your make process not producing coverage data the first time it is run, possibly due to cmake not correctly settings that variable you pass in.

I'll keep poking around.

chrisosaurus commented 6 years ago

Confirmed that the single build isn't producing the profile data

        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354549/log.txt

output

travis_time:end:014a2c6d:start=1531020384350424558,finish=1531020386367779703,duration=2017355145

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:12bf641b
$ find . -iname '*.gcda' | wc -l
0

travis_time:end:12bf641b:start=1531020386374362869,finish=1531020386390520461,duration=16157592

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:03a20972
$ find . -iname '*.gcno' | wc -l
1

running just ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354666/log.txt

outputs

travis_time:end:0557cb6e:start=1531020363849677574,finish=1531020365103355512,duration=1253677938

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:13f0336e
$ find . -iname '*.gcda' | wc -l
0

travis_time:end:13f0336e:start=1531020365110018399,finish=1531020365127668255,duration=17649856

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:01fb3190
$ find . -iname '*.gcno' | wc -l
1

running cmake/make/ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354576/log.txt

outputs

travis_time:end:0771da55:start=1531020637657257692,finish=1531020639721292131,duration=2064034439

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:1eb72f18
$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1eb72f18:start=1531020639731115174,finish=1531020639750849314,duration=19734140

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:1a6234f4
$ find . -iname '*.gcno' | wc -l
172

So the problem looks to be that your build tooling requires running cmake/make/ctest twice before it produces the gcno and gcda files.

Cpp-coveralls doesn't appear to be the problem here.

chrisosaurus commented 6 years ago

So poking around in CMakeFiles/ with grep looking for 'coverage' we get

single run of everything

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359763/log.txt

output

travis_time:end:1060f428:start=1531023482629119561,finish=1531023484216188718,duration=1587069157

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:076beae1
$ find . -iname '*.gcda' | wc -l
0

travis_time:end:076beae1:start=1531023484225581695,finish=1531023484243654484,duration=18072789

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:015e15a6
$ find . -iname '*.gcno' | wc -l
1

travis_time:end:015e15a6:start=1531023484255969002,finish=1531023484273764591,duration=17795589

The command "find . -iname '*.gcno' | wc -l" exited with 0.
travis_time:start:308de1c4
$ grep -Rni 'coverage' CMakeFiles/* | wc -l
2

So no gcno, no gcda, and only 2 mentions of 'coverage'

running cmake twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359814/log.txt

output

travis_time:end:0ce384e0:start=1531023535870351025,finish=1531023537691723347,duration=1821372322

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:1e795360
$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1e795360:start=1531023537698709357,finish=1531023537714924357,duration=16215000

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:0f774228
$ find . -iname '*.gcno' | wc -l
172

travis_time:end:0f774228:start=1531023537721351873,finish=1531023537738222833,duration=16870960

The command "find . -iname '*.gcno' | wc -l" exited with 0.
travis_time:start:0dd3c07f
$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

3 mentions of coverage, 54 gcda files, 172 gcno files

running 'everything' twice inside docker

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359864/log.txt

travis_time:end:0599ce52:start=1531023759612471574,finish=1531023761327501590,duration=1715030016

The command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.
travis_time:start:1f4625f9
$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1f4625f9:start=1531023761335501133,finish=1531023761353031077,duration=17529944

The command "find . -iname '*.gcda' | wc -l" exited with 0.
travis_time:start:134c59f5
$ find . -iname '*.gcno' | wc -l
172

travis_time:end:134c59f5:start=1531023761360645133,finish=1531023761376400031,duration=15754898

The command "find . -iname '*.gcno' | wc -l" exited with 0.
travis_time:start:2519013c
$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

Again 3 mentions of coverage, 54 gcda, 172 gcno files.

chrisosaurus commented 6 years ago

So at the very least, your cmake config is buggy, as these two commands produce different outputs:

cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest

cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./ && make && make tests && ctest

chrisosaurus commented 6 years ago

I have filed https://github.com/GlPortal/RadixEngine/issues/175 for the cmake config issue

chrisosaurus commented 6 years ago

I'm still seeing a lot of noise in the genereated output, lots of mentions for system included files which will throw your coverage stats way off.