Easy to use build and package tool for a custrom forks of ros2

[jmachowinski]

ROS 2 Production Task Proposal

Proposal Description:

There is a common need to build and package custom ros2 distributions. E.g. If you use ROS2 and hit a bug in rclcpp and you can't wait for the upstream merge of the fix etc.

Therefore the proposal is to come up with a build tool, where one can put in something like this file https://raw.githubusercontent.com/ros2/ros2/iron/ros2.repos that produces packages for each entry in the file.

The build tool is supposed be run on privately hosted infrastructure.

Ideally this tool should be extendible to package non public packages as well.

Estimated Effort:

No idea

Area of Impact:

CICD

Related Works:

bloom https://github.com/ros2/ci

[devrite]

We had one that integrated as verbs with colcon and used bloom. Instead we switched to buildfarm based pipeline generator for debían packages on Ubuntu in Gitlab.

Although the old efforsts could be reactivated. The new approach was scalable and could be parallelized to multiple runners/builders. For a colcon integrated tool you would replace the actual build steps with bloom and add any custom flags if needed to the generated build files. You can build your tasks the same ways as colcon does to keep the order and finish by installing the built package. We also had a version for ros1 with catkin. Ideally one would call the verb build_package. Alternatively, you let colcon build it as usual but use the installed files to build your package.

In our pipeline approach ideas to extend it exist, but depend on need and time and we had no urgent need so far.

In short: It checks if dependencies were updated and can set the level for the check to scale between e.g., daily/triggered rebuilds (ROS level) or weekly monthly (all dependencies).

What I would propose is that I remove our current stuff and make a Gitlab template out of it and then start rewriting it so it can be included or downloaded to your pipeline

[doganulus]

There is a common need to build and package custom ros2 distributions. E.g. If you use ROS2 and hit a bug in rclcpp and you can't wait for the upstream merge of the fix etc.

I would first ask why we do not build our fixed rclcpp fork from the source and use it directly in our environment. Why must it be a custom ROS 2 distro requiring much more effort? (assuming it is not an ABI/API changing fix, otherwise yes.)

Perhaps this is related to the fact that rosdep does not support versioned dependencies. I think that would not be an issue if we could specify rclcpp version in package dependencies where we could state our fork directly--probably from the source.

Any interest in reworking package dependencies?

[devrite]

Regarding reworking or extending dependency management in ROS:

Would be nice, but do not forget that also other package manager or native managers and build systems have their limitations. The question that comes to mind is that if you reimplement the features of another manager why not use it as the main one? I would track extending or reworking package management (or package definitions) to an own issue.

If we look at conan, conda, yocto/poky/bitbake and many others these already implement a lot of these details or have tools for management and distributing a custom distro. But it would be a completely other round that current ROS package definitions.

The issue I currently see with the ros package system (package.xml) is that it lacks in features and most people do not adhere to API/ABI versioning (SO versions package name and version in case of breaks). This also plays into generating automatic change reports that was mentioned in the last meeting. But the current package.xml spec allows to specify versions for dependencies. But I have never seen actual examples using it (not even myself) and how they would be respected by all ros-tooling. We need to be careful at least to not reinvent the wheel if it would be just better to use another package manager and build system or at least could just extend bloom with additional generators and features.

So bloom is already mighty tool but has it limits too, which can make it more cumbersome to manage custom distros too as well as generating custom build configs.

I would like another issue for what we would like the package definitions to hold and what the tools themselves are lacking currently (a CI/CD like ours would just use the tools that are easier to do the task).

Regarding CI/CD and native packaging:

This is the relevant issue here and there can be multiple approaches. Building pure workspaces with custom fixes of package is of course already supported by colcon and rosdep (and native/binary packages). We also use workspaces instead of packages quite often but we also like to have a ready to install set of packages for easy use.

Our approach is to generate a pipeline (Gitlab, Ubuntu, based on ros_buildfarm):

• Declare packages (to build) in a yaml and target architecture (currently needs runners for each)
• Declare custom dependencies (non-ros packages) in a yaml (general + arch wise).
• Check dependencies and order, check if dependencies changed (including not built ones) filter those which do not need updates
• Update distro file
• Prepare env (accepting any licenses or env vars, add apt repos)
• Create source debian, including buildinfo (used to check by subsequent calls) (custom version string to make custom built packages identifiable and overridden packages newer than a package of the same version).
• Build source debians
• Upload debians and sign

Since we use rosbuildfarm but wanted to avoid the dockerfile per package as well as gbp repos, we just fake a gbp repo and build the packages directly and can use a squiddebproxy for caching if needed (depends on your runner config if the stored packages will be kept). Also we don't do any testing here it really is just the build-packaging for now.

So if there is interest I would put the current version out on our or the public gitlab instance for testing and comments.

For reference: Conda: https://robostack.github.io/GettingStarted.html Yocto/poky/bitbake: https://github.com/ros/meta-ros And Alberto's work: https://github.com/irobot-ros/conan-ros2

Just in case we want to extend or reuse tooling there or even make their lives easier, when we add features to the other tooling.

[jmachowinski]

I would first ask why we do not build our fixed rclcpp fork from the source and use it directly in our environment. Why must it be a custom ROS 2 distro requiring much more effort? (assuming it is not an ABI/API changing fix, otherwise yes.)

Multiple reasons:

• If you modify rclcpp, you need to also rebuild about ~80 standard ros packages, as they depend on rclcpp. E.g. rviz2
• Note this proposal is about building and packaging. We are talking about production systems here, that we want to deploy at customers. In these systems, you only want binaries, not sources. Second, you want these binaries, to come from a defined reproducible environment (keyword reproducible builds).
• As to why packaged and installed, if the customer comes back to you with a bug, you need a way to figure out what software version caused the bug. E.g. you need to do something like query apt for the installed version.

If you look at this problem of building and packaging from an abstract perspective, it is:

• I have a base image
• I got a workspace
• I want to install all dependencies of my workspace
• I want to build it
• I want the build artifacts packaged (e.g. produce debian packages)

The interesting thing here that I want to point out is

• A ros2 distribution is just a workspace
• Your robot software is also just a workspace, with dependencies

And coming back to what I just wrote above, you do not only want your ros2 distro packaged, but also your application stack on top of it.

Therefore it might be possible to create a solution that solves/eases both problems for us.

One question that is still open to me is the granularity of the approach. We could either

• Build the whole workspace and pack the build artifacts as one big package
• Build only a package of the workspace at a time and only package this

[devrite]

We did the latter before as integration of catkin/colcon. Packaging the workspace as, a big package is already easy. The difficult thing is to track outside dependencies. If you do not need that a tar of basically install would already suffice.

Packaging each package of a WS can ve done by basically using bloom or ros infrastructure scripts as a verb to colcon. This will keep track of any specified xml specified dependencies.

For cmake based packages we could also exploit the inherent packaging and define cmake addons that could allow that but for python you would need a different approach

If packages were to use SO version for libs you would even be let debian helper check all SOs and auto add these with appropriate version requirements.

[artivis]

Hello, Regarding the 'building' aspect of this ticket, I though the following colcon extension would be of interest to you, at least for reference: colcon-in-container. The project actually contains 2 colcon extensions providing the build and test functionalities -in-container. The build-in-container extension,

• spawns a fresh linux container (lxc)
• mounts colcon's target (optionally, not the entire workspace, depending on options passed to colcon)
• installs the build deps (rosdep)
• builds everything
• copy the artifacts back on the host machine.

The test-in-container extension then,

• spawn a fresh container
• mounts colcon's target and the build artifacts
• installs the test deps
• execute the test target

edit: forgot to mention that one of the implication of this project is that you only need colcon, colcon-in-container and lxd installed on the host machine to build your ROS project. And that for any supported ROS distro (e.g. build a Foxy project on a 22.04 host)

Check out the repo for more info and do feel free to share any feedback.

[jspricke]

For smaller projects this should be doable with https://github.com/jspricke/ros-deb-builder-action (there are a number of forks adopting it to bigger build farms).

[devrite]

@jspricke Nice. Especially with sbuild. We usually just directly called the generated build files. Do you know, can you add extra arguments (e.g., for cmake) to sbuild (besides using custom build steps or environment variables)?

What we usually did for custom workspace build is to inject them into the generated debian build files and later even integrated the build into catkin/colcon to exploit the parallel builds and (config possibilities.

Also your approach can serve as a simple alternative to our approach exploiting the ros build farm/infrastructure scripts. You can generate a pipeline based on the yaml and individual steps you created, also exploiting parallelism.

I'llt try to upload our solution to Gitlab this week before my long holidays. If there is enough interest we can provide gh actions too but our internal solution is Gitlab.

Besides, everything we find adaptable we can try to create PRs/repos for https://github.com/ros-tooling/ like https://github.com/ros-tooling/action-ros-ci.

[jspricke]

@devrite not sure what you want to modify but you can surely do that. Also note that the Action bumps the compat level so it does parallel builds already (https://github.com/jspricke/ros-deb-builder-action/blob/main/build#L113).

[devrite]

@jspricke I mean package parallel as with colcon or multi job/runner.

[devrite]

Probably after my holiday. It is currently too loaded. Maybe my colleague works on it in the meantime.