Jenkins: No test report files were found. Configuration error?

[HannesSommer]

There seems to be a problem affecting some repositories. At least: aslam_nonparametric_estimation (see e.g. http://129.132.38.183:8080/job/aslam_nonparametric_estimation/254/consoleText - at the very end) aslam_simulation aslam_map_manager ??

I believe this is a common reason for those not to become green anymore!

Where do we put build server related issues?

[HannesSommer]

I think the lines like the following are related: cp: cannot stat `/var/lib/jenkins/jobs/aslam_nonparametric_estimation/workspace/build/sm_python/test_results/sm_python': No such file or directory Maybe there is something wrong with the CMakeLists.txt? But that affects then multiple packages..

[simonlynen]

Bugs related to the build-server go here: https://github.com/ethz-asl/continuous_integration

[simonlynen]

Yes the build-script tries to copy the test results to a common folder where they are parsed by jenkins. Can you verify that the results are copied to build/package_name/test_results/package_name on your machine?

[simonlynen]

In case there are no tests the jenkins JUnitParser plugin fails if it is configured to parse tests.

[HannesSommer]

Thanks for the ideas. Yes most, of these packages do the right thing on my machine (putting test results in build/package_name/test_results/package_name). Others don't have tests. But the problem seems to be something different. My current best guess is there is a conflict between the the FindGlog.cmake e.g. in ethzasl_brisk/brisk and the glog_catkin package. Resulting in many CMake warnings, wenn the build files are generated, like the following one:

[aslam_simulation] CMake Warning at /var/lib/jenkins/ros_catkin_ws/install_isolated/share/catkin/cmake/test/gtest.cmake:46 (add_executable):
[aslam_simulation]   Cannot generate a safe runtime search path for target
[aslam_simulation]   aslam_cv_serialization_tests because there is a cycle in the constraint
[aslam_simulation]   graph:
[aslam_simulation] 
[aslam_simulation]     dir 0 is [/var/lib/jenkins/jobs/aslam_simulation/workspace/devel/lib]
[aslam_simulation]       dir 1 must precede it due to runtime library [libglog.so.0]
[aslam_simulation]     dir 1 is [/usr/local/lib]
[aslam_simulation]       dir 0 must precede it due to runtime library [libglog.so.0]
[aslam_simulation]     dir 2 is [/var/lib/jenkins/ros_catkin_ws/install_isolated/lib]
[aslam_simulation] 
[aslam_simulation]   Some of these libraries may not be found correctly.
[aslam_simulation] Call Stack (most recent call first):
[aslam_simulation]   dependencies/aslam_cv/aslam_cv_serialization/CMakeLists.txt:22 (catkin_add_gtest)

(to be found e.g. in http://129.132.38.183:8080/job/aslam_nonparametric_estimation/246/consoleText)

I believe it is that because the tests seem to be just not written as build files after those warnings. And it seems to be the case that currently that way many packages get not tested at all (on the build server) and that only those repositories that contain packages that have tests run are green. It doesn't matter that most of them have other packages that do not get there tests executed (that could be a flaw of the jenkins setup).

Does that sound plausible to you? I'm unsure how to proceed best.

[simonlynen]

I can take a look tomorrow and see how I can resolve that.

I am pretty sure the warnings are not related though. On Jun 25, 2014 3:12 AM, "HannesSommer" notifications@github.com wrote:

Thanks for the ideas. Yes most, of these packages do the right thing on my machine (putting test results in build/package_name/test_results/package_name). Others don't have tests. But the problem seems to be something different. My current best guess is there is a conflict between the the FindGlog.cmake e.g. in ethzasl_brisk/brisk and the glog_catkin package. Resulting in many CMake warnings, wenn the build files are generated, like the following one:

[aslam_simulation] CMake Warning at /var/lib/jenkins/ros_catkin_ws/install_isolated/share/catkin/cmake/test/gtest.cmake:46 (add_executable): [aslam_simulation] Cannot generate a safe runtime search path for target [aslam_simulation] aslam_cv_serialization_tests because there is a cycle in the constraint [aslam_simulation] graph: [aslam_simulation] [aslam_simulation] dir 0 is [/var/lib/jenkins/jobs/aslam_simulation/workspace/devel/lib] [aslam_simulation] dir 1 must precede it due to runtime library [libglog.so.0] [aslam_simulation] dir 1 is [/usr/local/lib] [aslam_simulation] dir 0 must precede it due to runtime library [libglog.so.0] [aslam_simulation] dir 2 is [/var/lib/jenkins/ros_catkin_ws/install_isolated/lib] [aslam_simulation] [aslam_simulation] Some of these libraries may not be found correctly. [aslam_simulation] Call Stack (most recent call first): [aslam_simulation] dependencies/aslam_cv/aslam_cv_serialization/CMakeLists.txt:22 (catkin_add_gtest)

(to be found e.g. in http://129.132.38.183:8080/job/aslam_nonparametric_estimation/246/consoleText )

I believe it is that because the tests seem to be just not written as build files after those warnings. And it seems to be the case that currently that way many packages get not tested at all (on the build server) and that only those repositories that contain packages that have tests run are green. It doesn't matter that most of them have other packages that do not get there tests executed (that could be a flaw of the jenkins setup).

Does that sound plausible to you? I'm unsure how to proceed best.

— Reply to this email directly or view it on GitHub https://github.com/ethz-asl/programming_guidelines/issues/21#issuecomment-47083112 .

[HannesSommer]

Hm, I agree these warnings don't sound that severe but I could not find any other explanation why the tests are missing in the makefiles even though they are in the CMakeLists.txt. And that they do miss you can find in the same log where things like :

[build] Runtime: 1 minute and 0.9 seconds
*** sourcing the devel space
*** aslam_splines -- Attempting to run unit tests.
*** aslam_splines -- No unit tests found
*** aslam_splines -- Copy unit test results to common folder.
cp: cannot stat `/var/lib/jenkins/jobs/aslam_nonparametric_estimation/workspace/build/aslam_splines/test_results/aslam_splines': No such file or directory

can be found. Looking at the source code of the build scripts (https://github.com/ethz-asl/continuous_integration/blob/master/catkin_make_repo#L40-L55) I can only conclude that the Makefiles are missing the run_tests target.

[HannesSommer]

I would like to make some experiments to track that bug down. Unfortunately I have reduced rights on Jenkins (cannot see or modify configuration or run builds with different command line - that would be what I need). @simonlynen could you grand me those rights for at least either aslam_optimization or some test package, if we have one?

[furgalep]

Hannes, what is your Jenkins User Name?

[HannesSommer]

Should be HannesSommer.

[furgalep]

Okay, I tried to add you.

[HannesSommer]

Thanks :)

[HannesSommer]

Worked :)!

[HannesSommer]

Ok, worked a bit on that.

The warnings I hoped to be related to the cause seems to be unrelated. The warning is cause by the fact that brisk is fining its own libglog.so in /usr/local/lib while the packages using glog_catkin take the one in devel/lib. I could hack around it by changing temporarily the FindGlog.cmake in brisk to also find the one in devel/lib - just to check whether it helps. Apparently it doesn't. So Simon was right with doubting that it is related!

I could verify my theory about what is failing on the second highest level: the Makefile in build/PACKAGENAME is missing the run_tests target when the problem occurs. Furthermore the build/PACKAGENAME/CMakeFiles/Makefile2 is missing, which is used by the run_tests target. However the run_tests_PACKAGENAME targets in the first Makefile are still there! Funnily the problem apparently only affects the packages being directly tested - not its dependencies. There the run_tests targets get written.

So it is true that the problem is the CMake phase, actually producing incomplete makefiles!

I've found a way to recover from that : remove all the build/PACKAGENAME folders for the primary packages in that repository :) - a single one could be not enough to fix itself (probably it is caused by some for all). But I have still no idea what is actually the problem. Removing the build/CMakeCache.txt does not help and is not necessary!

Looks like some wired catkin/cmake bug to me. We will see if that happens again or is solved in the meantime. I will fix the current packages by remove the corresponding build folders.

[furgalep]

Resolved?

[HannesSommer]

Probably yes. I know how to solve it (by deleting files). I don't know how it happened. But I have not seen it happen since then anymore. My best guess: it was a bug in an old catkin version.