Noetic binary builds failing on the buildfarm

[gavanderhoorn]

Looks like none of the binary builds have succeeded since this packages was released: Nbin_uF64__opw_kinematicsubuntu_focal_amd64binary. See also status_page/ros_noetic_default.html?q=opw.

Source build only succeeded since the 0.4.0 release: Nsrc_uFopw_kinematics__ubuntu_focalsource. Not sure what changed there.

Example log from the failed binary build:

23:17:45 -- Building with lcov Code Coverage Tools
23:17:45 CMake Warning at /opt/ros/noetic/lib/cmake/ros_industrial_cmake_boilerplate/code_coverage.cmake:153 (message):
23:17:45   Code coverage results with an optimized (non-Debug) build may be misleading
23:17:45 Call Stack (most recent call first):
23:17:45   CMakeLists.txt:20 (initialize_code_coverage)
23:17:45 
23:17:45 
23:17:45 CMake Error at /opt/ros/noetic/lib/cmake/ros_industrial_cmake_boilerplate/code_coverage.cmake:165 (message):
23:17:45   lcov not found! Aborting...
23:17:45 Call Stack (most recent call first):
23:17:45   CMakeLists.txt:20 (initialize_code_coverage)
23:17:45 
23:17:45 
23:17:45 -- Configuring incomplete, errors occurred!
23:17:45 See also "/tmp/binarydeb/ros-noetic-opw-kinematics-0.4.0/obj-x86_64-linux-gnu/CMakeFiles/CMakeOutput.log".
23:17:45    cd obj-x86_64-linux-gnu && tail -v -n \+0 CMakeCache.txt

From the trace, it seems ros_industrial_cmake_boilerplate is not finding lcov and then bailing (which is probably because it's only listed as a test_depend).

Should initialize_code_coverage() only be called within if (OPW_ENABLE_TESTING)?

[gavanderhoorn]

Missing lcov tracked in https://github.com/ros-industrial/ros_industrial_cmake_boilerplate/issues/41.

[Levi-Armstrong]

Should initialize_code_coverage() only be called within if (OPW_ENABLE_TESTING)?

I believe so. Do you think that should solve the build issue?

[gavanderhoorn]

Only partially I believe.

I can't find it right now, but I'm not sure when the buildfarm will consider test_depend dependencies for installation.

Who would have to set OPW_ENABLE_TESTING?

Is there a reason you're not using vanilla CMake ENABLE_TESTING?

---

Edit: you could run a prerelease test to see whether your package builds successfully. The site should allow you to generate a command with multiple source repositories.

[Levi-Armstrong]

Is there a reason you're not using vanilla CMake ENABLE_TESTING?

It was more for convenience because if I have all the repositories in one workspace and set the ENABLE_TESTING it would build everything. This allow for easily building just a given repositories without having to manually specify all of the package in the command line. Though I could add both if where if either are set it builds the tests if that is preferred.

[gavanderhoorn]

Could be related: https://github.com/ros-infrastructure/ros_buildfarm/issues/870 (but I remember other discussions about this).

[Levi-Armstrong]

@gavanderhoorn I have a question related to to the build farm. I submitted a new version of ros_industrial_cmake_boilerplate and opw_kinematics with the changes to only enable code coverage when the compiler flag is set. Since both are submitted when they build on the buildfarm will opw_kinematics use the new version submitted or will it use the previously released version?

[gavanderhoorn]

The buildfarm starts builds for various reasons, one being new releases, but also new releases of dependencies.

So even if opw_kinematics gets build before ros_industrial_cmake_boilerplate and fails (because it's trying to use the old version), the next successful build of ros_industrial_cmake_boilerplate should trigger a build of all its dependents, at which point opw_kinematics should get built again.

[Levi-Armstrong]

@gavanderhoorn I just ran the pretest with the latest opw_kinematics and ros_industrial_cmake_boilerplate without any errors. Though it did not run any tests because you have to set OPW_ENABLE_TESTS. Should I update the enable test to be: if (OPW_ENABLE_TESTING OR ENABLE_TESTING)?

[Levi-Armstrong]

Also it is still failing on the build farm but the pretest is passing. I believe it may be related to the build farm issue you provided, though I would have expected the pretest to also fail. Should I switch these depends to build_depend for now? Unfortunately it will require the install of these depend even if they are not intended to be used.

[Levi-Armstrong]

The buildfarm starts builds for various reasons, one being new releases, but also new releases of dependencies.

So even if opw_kinematics gets build before ros_industrial_cmake_boilerplate and fails (because it's trying to use the old version), the next successful build of ros_industrial_cmake_boilerplate should trigger a build of all its dependents, at which point opw_kinematics should get built again.

Thank you for the clarification.

[Levi-Armstrong]

@gavanderhoorn Everything is now building when going through the pretest process. The ros_industrial_cmake_boilerplate has successfully build on Jenkins but it appears that opw_kinematics is still leveraging the old version of ros_industrial_cmake_poilerplate. I assume at some point it will get triggered to build with the new version.

[Levi-Armstrong]

It looks like now it is failing due to compiler option -mno-avx on arm. I have a PR to disable this option.

[Levi-Armstrong]

@gavanderhoorn I believe the issue may be resolved now but it is unclear how to best check this. Is there any documentation on how to track your packages progress when releasing on the buildfarm? One on the status page is still marked missing so I assume that is not a failure just that it has not been synced yet; is that correct?

[gavanderhoorn]

For Noetic the table basically states that there were successful source builds for the current version and the previous one, but the binary build only succeeded for the latest release.

The last square is red because a sync is indeed needed to get the build artefacts from the last successful build into the main repositories.

---

Edit: I don't know whether you'd call it documentation, but the ROSCon 2016 Seoul Day 2: The ROS build farm - what it can do for me (slides) presentation goes into some detail on how it all works.

Slide 21 goes over the status page and the columsn there.

[Levi-Armstrong]

@gavanderhoorn It looks like everything is green now. Should this be closed?

[gavanderhoorn]

Yes, seems like it's all resolved.

thanks @Levi-Armstrong :+1: