Issues with rosidl generators [Urgent]

[SteveMacenski]

http://build.ros2.org/job/Cbin_uB64__nav2_navfn_planner__ubuntu_bionic_amd64__binary/7/console

16:08:01 CMake Error at /opt/ros/crystal/share/nav2_tasks/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_package):
16:08:01   By not providing "Findrosidl_default_generators.cmake" in CMAKE_MODULE_PATH
16:08:01   this project has asked CMake to find a package configuration file provided
16:08:01   by "rosidl_default_generators", but CMake did not find one.
16:08:01 
16:08:01   Could not find a package configuration file provided by
16:08:01   "rosidl_default_generators" with any of the following names:
16:08:01 
16:08:01     rosidl_default_generatorsConfig.cmake
16:08:01     rosidl_default_generators-config.cmake
16:08:01 
16:08:01   Add the installation prefix of "rosidl_default_generators" to
16:08:01   CMAKE_PREFIX_PATH or set "rosidl_default_generators_DIR" to a directory
16:08:01   containing one of the above files.  If "rosidl_default_generators" provides
16:08:01   a separate development package or SDK, be sure it has been installed.
16:08:01 Call Stack (most recent call first):
16:08:01   /opt/ros/crystal/share/nav2_tasks/cmake/nav2_tasksConfig.cmake:38 (include)
16:08:01   CMakeLists.txt:17 (find_package)
16:08:01 

[SteveMacenski]

Every failed job except for these 2 failed from this error. We need to resolve this to get most of the packages out

http://build.ros2.org/job/Cbin_ubv8_uBv8__nav2_tasks__ubuntu_bionic_arm64__binary/9/console http://build.ros2.org/job/Cbin_ubv8_uBv8__nav2_util__ubuntu_bionic_arm64__binary/9/console Which seem to just be from a timeout (?)

[SteveMacenski]

LIst of failed jobs in red at these 2 links

http://build.ros2.org/job/Cbin_uB64__navigation2__ubuntu_bionic_amd64__binary/ http://build.ros2.org/job/Cbin_ubv8_uBv8__navigation2__ubuntu_bionic_arm64__binary/

[mkhansenbot]

Steve, I'm not sure what's happening yet as I'm unfamiliar with the build farm flow. It appears to be dying during the "Run Dockerfile" step and looks like the timeout is due to being unable to open a URL:

15:49:37   File "/usr/lib/python3/dist-packages/rosdistro/loader.py", line 48, in load_url
15:49:37     fh = urlopen(url, timeout=timeout)

I don't know why this is timing out. I don't believe it has anything to do with our code. Maybe @nuclearsandwich knows?

[nuclearsandwich]

I don't know why this is timing out. I don't believe it has anything to do with our code. Maybe @nuclearsandwich knows?

The timeout is an infrastructure issue and shouldn't to my knowledge be affected by any package contents.

The CMake failure during package builds is something I've proposed a solution to in #474

[mkhansenbot]

@nuclearsandwich - that wasn't clear to me from your description in #474

It actually looks to me like the timeout issue was resolved in a later build, but ther real issue is cross compiling from ARM64. http://build.ros2.org/job/Cbin_ubv8_uBv8__nav2_amcl__ubuntu_bionic_arm64__binary/9/console#console-section-0

Is that what #474 is addressing?

[nuclearsandwich]

@nuclearsandwich - that wasn't clear to me from your description in #474

My apologies. I shouldn't open PRs before coffee.

[nuclearsandwich]

It actually looks to me like the timeout issue was resolved in a later build, but ther real issue is cross compiling from ARM64. http://build.ros2.org/job/Cbin_ubv8_uBv8__nav2_amcl__ubuntu_bionic_arm64__binary/9/console#console-section-0

Is that what #474 is addressing?

I don't believe cross-compilation is an issue here since these builds run on native ARM hosts (or in QEMU with a native ARM image if no ARM host is available). Nothing on the ROS buildfarm should cross compile without the package author working very hard to achieve it.

The binary jobs all have errors like this:

23:16:28 CMake Error at /opt/ros/crystal/share/nav2_tasks/cmake/ament_cmake_export_dependencies-extras.cmake:21 (find_package):
23:16:28   By not providing "Findrosidl_default_generators.cmake" in CMAKE_MODULE_PATH
23:16:28   this project has asked CMake to find a package configuration file provided
23:16:28   by "rosidl_default_generators", but CMake did not find one.

This error message shows that the CMake file /opt/ros/crystal/share/nav2_tasks/cmake/ament_cmake_export_dependencies-extras.cmake which is installed by the nav2_tasks package is trying to find_package(rosidl_default_generators) which isn't being installed by the specific packages being built since it's not listed as a dependency.

I took a cursory glance at the nav2_tasks package and didn't see anything in its operation that suggested its downstream packages actually do need rosidl_default_generators installed or any of its other dependencies, all of which were exported prior to #474.

[SteveMacenski]

@nuclearsandwich this all seems reasonable to me, thanks for taking a more hands on role in this first release cycle. That issue was pretty subtle to me so I'm glad it jumped out to you :-)

Is there a way we can run this in CI so that these issues appear to us? I dont see that colcon has an isolated build option

[nuclearsandwich]

Is there a way we can run this in CI so that these issues appear to us? I dont see that colcon has an isolated build option

Colcon runs all builds isolated by default, analogous to catkin_make_isolated. But neither build tool isolates system dependencies. So the isolated devel and PR jobs on the ROS buildfarm can catch dependency hygiene issues between packages in the same repository. To completely check dependencies you could build each package in the repository in its own container (or other system-dependency isolating device) with only dependencies specified by that package.xml. I don't know of any tools to automate that, especially factoring in making sure that inter-dependent packages in the same repository have the results of their isolated builds shared to downstream packages hermetically.

The binary jobs do build each package in its own container which is why that's where these issues are generally first encountered but those jobs can only be run after a bloom release is made (it doesn't have to be a bloom release to an upstream rosdistro but there's a lot of moving parts to creating a fake rosdistro for one-off binary jobs). Even in ROS 2 core there is usually some cleanup of dependencies required at release time (A recent example). I've just been whittling them down since Beta 2 :grin:

[SteveMacenski]

Glad to hear its not just us :smiley_cat: releasing broken stuff makes my stomach churn a little. I'll have to noodle on this for a little while if it becomes a reoccurring theme after this release.

[nuclearsandwich]

I'll have to noodle on this for a little while if it becomes a reoccurring theme after this release.

I've had daydreams (no time to do more but dream) about extending bloom to enable a full continuous deployment pipeline so that packages can go from source commit -> bloom -> ros_buildfarm / rosdistro configurator -> ros_buildfarm -> *.debs with no manual steps. If that's something you're interested in we can find time to at least talk about it.

[SteveMacenski]

Sure, I'm always around. I'm up in SF but I certainly wouldn't mind a trip to OR to chat :+1: See where all the magic happens

[SteveMacenski]

oops prematurely closed

[mkhansenbot]

@SteveMacenski - I see you re-opened this are we failing again? Everything looks green to me.

[SteveMacenski]

Yes, just motion primatives though

17:05:15 Scanning dependencies of target motion_primitives
17:05:15 make[4]: Leaving directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/obj-x86_64-linux-gnu'
17:05:15 make -f CMakeFiles/motion_primitives.dir/build.make CMakeFiles/motion_primitives.dir/build
17:05:15 make[4]: Entering directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/obj-x86_64-linux-gnu'
17:05:15 [ 16%] Building CXX object CMakeFiles/motion_primitives.dir/src/spin.cpp.o
17:05:15 /usr/bin/c++   -I/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/include -I/opt/ros/crystal/include -I/usr/include/eigen3  -g -O2 -fdebug-prefix-map=/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4=. -fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2   -Wall -Wextra -Wpedantic -Werror -std=gnu++14 -o CMakeFiles/motion_primitives.dir/src/spin.cpp.o -c /tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/src/spin.cpp
17:05:20 In file included from /tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/include/nav2_motion_primitives/spin.hpp:22:0,
17:05:20                  from /tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/src/spin.cpp:22:
17:05:20 /tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/include/nav2_motion_primitives/motion_primitive.hpp: In member function ‘nav2_tasks::TaskStatus nav2_motion_primitives::MotionPrimitive<CommandMsg, ResultMsg>::cycle(ResultMsg&) [with CommandMsg = geometry_msgs::msg::QuaternionStamped_<std::allocator<void> >; ResultMsg = std_msgs::msg::Empty_<std::allocator<void> >]’:
17:05:20 /tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/include/nav2_motion_primitives/motion_primitive.hpp:152:12: error: ‘status’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
17:05:20      return status;
17:05:20             ^~~~~~

17:05:22 cc1plus: all warnings being treated as errors
17:05:22 CMakeFiles/motion_primitives.dir/build.make:65: recipe for target 'CMakeFiles/motion_primitives.dir/src/spin.cpp.o' failed
17:05:22 make[4]: Leaving directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/obj-x86_64-linux-gnu'
17:05:22 make[4]: *** [CMakeFiles/motion_primitives.dir/src/spin.cpp.o] Error 1
17:05:22 make[3]: *** [CMakeFiles/motion_primitives.dir/all] Error 2
17:05:22 CMakeFiles/Makefile2:422: recipe for target 'CMakeFiles/motion_primitives.dir/all' failed
17:05:22 make[3]: Leaving directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/obj-x86_64-linux-gnu'
17:05:22 make[2]: *** [all] Error 2
17:05:22 Makefile:143: recipe for target 'all' failed
17:05:22 make[2]: Leaving directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4/obj-x86_64-linux-gnu'
17:05:22 dh_auto_build: cd obj-x86_64-linux-gnu && make -j1 returned exit code 2
17:05:22 debian/rules:34: recipe for target 'override_dh_auto_build' failed
17:05:22 make[1]: Leaving directory '/tmp/binarydeb/ros-crystal-nav2-motion-primitives-0.1.4'
17:05:22 make[1]: *** [override_dh_auto_build] Error 2
17:05:22 make: *** [build] Error 2
17:05:22 debian/rules:22: recipe for target 'build' failed
17:05:22 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
17:05:22 E: Building failed
17:05:22 Traceback (most recent call last):
17:05:22   File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 138, in build_binarydeb
17:05:22     subprocess.check_call(cmd, cwd=source_dir)
17:05:22   File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
17:05:22     raise CalledProcessError(retcode, cmd)
17:05:22 subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-crystal-nav2-motion-primitives']' returned non-zero exit status 1.
17:05:22 # END SUBSECTION

[SteveMacenski]

Looks like a straight forward fix though - an actual code error rather than dependencies magic :fireworks:

[cdelsey]

I see that this build is compiling with -O2 flag. I wonder if that is the difference on why we don't see this in our local/Travis builds.

[SteveMacenski]

Good catch - we should add that to Travis so we pick it up

[orduno]

I've pushed a revision to the PR. That should solve the problem with nav2_motion_primitives.

[SteveMacenski]

awesome