ROS Package Collections

[doganulus]

ROS 2 Production Task Proposal: Collections

Proposal Description:

This proposal targets multi-package ROS projects and aims to streamline build and dependency management for basic and advanced use cases. In the following, we call a multi-package ROS project a ROS package collection and treat it as a single entity to manage.

A ROS package collection is assumed to be developed on a single version-controlled online repository by a single organization. We assume that the <hostname>/<organization>/<collection_name> uniquely identifies the collection. The organization has full control over all packages in the collection.

The proposal places package collections between package and system levels in the hierarchy as follows:

• Node executable
• Package
• Collection
• System

The proposal includes a motivation for ROS package collections and defines a collection manifest specification that declares dependencies, environments, and other metadata at the collection level. Initially, we consider the following features under this task proposal issue:

• Multiple environment declarations per collection
  • default, runtime, build, devel, test, and developer-defined environments
• Multi-tool dependency declaration per environment
  • Explicit use of rosdep, apt, apt_repository, pip, cargo, cmake, shell, ...
  • Encourage the use of existing formats like requirements.txt and cargo.toml
• Override semantics for configuration options at global, environment, tool, and unit levels
• Third-party source repositories are integrated into the collection specification
• Single-file collection specification for basic use cases (top-level collection.yml)
• Multi-file collection specification for advanced use cases (top-level collection folder)
• Custom standardized script (verb) declaration per collection such as build, test

The proposal does not include a reference implementation. It initially aims to increase the understanding of production requirements, discuss semantics and mechanisms, and reach an easily implementable specification.

Estimated Effort:

(2 months) Requirements elicitation and schema specification

Area of Impact:

Build, testing, distribution, deployment.

Related Works:

• ROS package manifests 1, 2, 3
• ROS meta-packages
• ROS variants
• Devcontainers
• Ansible execution environments

[doganulus]

Collection Manifest

The proposal introduces top-level collection.yml file to specify environments, dependencies, and collection metadata.

The collection.yml is equivalent to package.xml at the collection level. The relation between package and collection manifests will be discussed later.

The full specification will be defined in JSON Schema. Below are some examples demonstrating the structure of collection manifests.

First example

# collection.yml

organization: "my_company"
name: "my_ros_collection"
description: "describe the collection"
version: "1"

depends:
  rosdep: # tool or package manager name is explicit 
    packages: [boost, other-rosdep-key] # list multiple rosdep keys here

Single collection file with multiple environments

# collection.yml

environments:
  runtime:
    depends: 
      # Declare collection runtime (exec) dependencies using rosdep keys
      rosdep: [...]

  build:
    extends: [runtime] # depends on everything the runtime environment depends on
    depends:
      # Declare build dependencies using rosdep keys
      rosdep: [...]

  devel:
    extends: [build] # depends on everything the build environment depends on
    depends:
      # Declare extra development dependencies using rosdep keys
      rosdep: [...] 

More advanced collection file with multiple environments

# collection.yml

environments:
  runtime:
    depends: 
      apt: 
        packages: [libboost-dev, libssl] 
      yum:
        packages: []
      pip:
        packages_file: requirements.txt # file spec
      cargo:
        packages_file: cargo.toml # file spec
      zypper:
        packages: []

  build:
    extends: [runtime]
    depends:
      apt: 
        packages: [g++, cmake]
        options: # Override global apt options
      cmake:
        # repositories functionality integrated here and expanded
        # fetch using git and build with cmake default options
        # Makefile or meson based repositories goes under their key
        repositories: 
          - repo: github.com/other_organization/other_repo
            dest: /tmp/
            version: main
            options:
              git: # Override global git options at unit level if desired
              cmake: # Override global cmake options at unit level if desired

    options:
      git: # Override global git options at environment level if desired
      cmake: # Override global cmake options at environment level if desired 

  # developers can define custom environments 
  my_devel_extra:
    container: 
      image: ros:humble # specify a development container image 
      # build: # or build a container locally <see docker.compose.services.build>
      #   dockerfile: ./Dockerfile
    extends: [build]
    depends: 
      pip: 
        packages: ["numpy>=1.4", "scipy==2.12"]

  # default environment is inherited automatically by others
  # top-level depends element resolves to environments.default.depends
  default: 
    depends: 
      apt: [] # list default apt dependencies here

# Top-level options will be passed to individual tools. 
# These options may be overridden at environment and unit levels if applicable.
options: 
  apt:
    update: true
  git:
    clone: true
    recursive: true
    depth: 1

# Top-level options will be passed to individual tools. 
# Similar to package.json/scripts elements
commands:
  build: colcon build
  test: colcon test

Multi-file collection specification

Production environments may require multiple and experimental environments with many customizations. In these cases, a multi-file layout can be more managable. The multi-file layout for collections are as follows:

.
|-- collection.yml
|-- environments
|   |-- default.yml
|   |-- runtime.yml
|   |-- build.yml
|   |-- devel.yml
|   |-- test1.yml
|   |-- test2.yml
|   |-- ...
|   |-- testN.yml
|
|-- packages
    |-- package_1/
    |-- package_2/

Multi-file environment specification

Even more...

.
|-- collection
|   |-- main.yml
|   |-- options.yml
|   |-- environments
|       |-- default
|           |-- main.yml
|           |-- options.yml
|           |-- Containerfile
|       |-- runtime
|           |-- main.yml
|           |-- options.yml
|           |-- Containerfile
|       |-- build
|           |-- main.yml
|           |-- options.yml
|           |-- requirements.txt
|           |-- Containerfile
|       |-- devel
|           |-- main.yml
|           |-- options.yml
|           |-- requirements.txt
|           |-- Containerfile
|
|-- packages
    |-- package_1/
    |-- package_2/

[artivis]

This is an interesting proposal altho it's scope seem rather large.

Could you provide a concrete example where the existing tooling would come short but is covered by this package collection declaration? Beside the support of more packages managers and that of their respective packages list declaration format, a fair share can already be achieved with the existing tooling (rosdep, vcstool, colcon, etc) -- admittedly with some scripting tho. Some of the info in the collection file(s) seems to be redundant with the existing package.xml. Ideally, how would that work here?

For reference, I just came across the casey/just tool which, for a part, and in conjunction with the existing tooling, might be what you are looking for.

[doganulus]

Could you provide a concrete example where the existing tooling would come short

1. Rosdep does not respect dependency versions. Here, we may use the package manager directly and they support package versions normally.
2. No need to add every system package as a rosdistro key. The proposal gives ROS users an option to manage their own dependencies directly. This also reduces the burden on ROS infrastructure and maintenance.
3. The vcstool tool does not support single_branch, depth, or other options by repo-basis. The proposal gives users an option to manage their source dependencies more directly.
4. Dependencies are managed collectively at the collection level. This reduces the number of lines for dependency declarations. Fewer things to update.
5. The native support to declare multiple custom environments is a big plus for production.
6. The proposal encourages to use native-tooling. Reusing requirements.txt or cargo.toml is encouraged instead of repeating dependencies in the manifest. This gives a native installation path for pure Python and pure Rust ROS packages.
7. The proposal gives more flexibility to source dependencies and unifies the current practice of vcstool+colcon but allows others such as git+cmake or git+meson.
8. It is easier to declare custom deb, snap, or flatpak packages in this manifest.
9. It supports shell dependencies to execute arbitrary scripts as a fallback for every advanced use case not covered in the specification.
10. Finally, this is designed with containers in mind. We must be able to build environments as containers.

Some of the info in the collection file(s) seems to be redundant with the existing package.xml. Ideally, how would that work here?

Yes, I need more opinions on how to handle metadata at package and collection level. I think, in most cases, declaring them at the collection level would be sufficient but not sure.

For reference, I just came across the casey/just tool which, for a part, and in conjunction with the existing tooling, might be what you are looking for.

It is an interesting tool, yet this proposal is only about the declarative specification. There might be different approaches to implementation (mine would be Ansible). But any compliant tool must compile each environment declaration into a sequence of CLI commands and then execute. The commands section can use make or just if the developer wishes. It is inspired from npm package format and no extra logic is there.

[doganulus]

Based on the feedback in the meeting, I wrote down the discussion points and my revised answers for future reference. Feel free to add any comments in the following.

We already use rosdep for dependency management.

A thought experiment. Assume I have a package depending on the 'boost' rosdep key and it resolves to v1.74 on Debian12/Ubuntu2204. And also assume my package uses a certain functionality removed in the following versions. When Jazzy arrives, the rosdep key will resolve to v1.83, which will break my package, create friction, and probably slow down the migration to Jazzy. But Noble continues to package v1.74, so we could continue to use v1.74 and migrate to Jazzy if we had used apt directly. This proposal, therefore, helps more gradual migrations.

And here is a real case study: https://github.com/ros/rosdistro/pull/40033#pullrequestreview-1920121202

We maintain our sources.list for unsupported/old keys.

But why this extra maintenance burden? Debian/Ubuntu/RHEL/SUSE devs are already doing it. I believe we should not spend our limited maintenance budget on such tasks. This proposal provides a way to use those package managers without rosdistro broker.

We already have 'exec', 'build', 'test' environment dependency declarations in package.xml.

True. But only those we have, and it's fixed. Production environments already require more customization. There might be different build/test/runtime environments for the same package, platform-wise (x86 or arm) or 3rd-party-library-wise (e.g. build with or without CUDA) or even just for tutorials to help newcomers with extra tools and libraries.

[peci1]

It seems to me creating a new toolset for this is a bit impractical.

There already is a name for a collection of ROS packages - "stack". So why create a new name?

Rosdep not supporting version specs is a thing that could (should) be fixed: https://github.com/ros-infrastructure/rosdep/issues/325 . Package.xml support is already there.

A lot of the proposed features could be handled by using env variables in package.xml conditional dependencies. This is a pretty much overlooked thing that has a lot of uses. E.g. <build_depend if="$ACME_ENV=='dev'">dev-only-dep</build_depend>.

What would a stack-level dependency mean exactly? Isn't it better if each package explicitly specifies its own dependencies?

Duplicating rosdep definition functionality seems unnecessary. A much better approach in my view would be to allow specifying package-level rosdep list files so that you don't need to upstream all dependencies to rosdistro and you can only maintain platforms of interest. Of course, such specifications would be prohibited on the public buildfarm. But generally, upstreaming the dependencies is not much work and once you upstream them, you lower your own maintenance burden (at the expense of rosdistro maintenance, but I assume rosdep keys are not too much of a burden anyways).

Allowing to specify how to build a 3rd party package from a metadata file is a way to hell... I'm strongly against that. If absolutely needed, rdmanifest can be used for this.

---

Detailed comments to your points:

1. Rosdep does not respect dependency versions. Here, we may use the package manager directly and they support package versions normally.

As said, a PR to rosdep would fix this.

2. No need to add every system package as a rosdistro key. The proposal gives ROS users an option to manage their own dependencies directly. This also reduces the burden on ROS infrastructure and maintenance.
3. It is easier to declare custom deb, snap, or flatpak packages in this manifest.

Having the option to specify custom rosdep lists in package.xml would fix this.

3. The vcstool tool does not support single_branch, depth, or other options by repo-basis. The proposal gives users an option to manage their source dependencies more directly.

A PR would fix this.

4. Dependencies are managed collectively at the collection level. This reduces the number of lines for dependency declarations. Fewer things to update.

Not sure this is useful. I prefer keeping dependencies as close to their use as possible.

5. The native support to declare multiple custom environments is a big plus for production.
6. Finally, this is designed with containers in mind. We must be able to build environments as containers.

Environment variables in package.xml can do that. Or you can explicitly declare the environments and a few of their properties in <export> section of the stack's package.xml.

6. The proposal encourages to use native-tooling. Reusing requirements.txt or cargo.toml is encouraged instead of repeating dependencies in the manifest. This gives a native installation path for pure Python and pure Rust ROS packages.

This would make it much harder to determine all dependencies of a ROS package. I understand it'd be nice to support other tools' native metadata definition files, but there's this clash. Maybe there could be some converter that would be able to read pyproject.toml etc. into a package.xml-like object?

7. The proposal gives more flexibility to source dependencies and unifies the current practice of vcstool+colcon but allows others such as git+cmake or git+meson.

As said earlier, I really don't like this line of thought... Rdmanifest seems to be a good middle way for me. You can even keep the "implementation" rdmanifest file with the package, as done e.g. in https://github.com/basler/pylon-ros-camera/blob/master/pylon_camera/rosdep/pylon_sdk.rdmanifest . In conjunction with package-defined custom rosdep lists, this would allow basically any custom dependency, but it still separates it so that the build logic is not a part of package.xml.

9. It supports shell dependencies to execute arbitrary scripts as a fallback for every advanced use case not covered in the specification.

Rdmanifest.

[doganulus]

A lot of the proposed features could be handled by using env variables in package.xml conditional dependencies. This is a pretty much overlooked thing that has a lot of uses. E.g. dev-only-dep.

This would not scale. The single file specification here would not scale either. Therefore, the proposal include multi-file, multi-directory environment specifications. From the single file to multi-directory, we must use a virtual document model (DOM).

Duplicating rosdep definition functionality seems unnecessary.

If we take one step back, it is the rosdep, which duplicates other package managers' functionality by wrapping it with a simple name lookup operation. Once this layer is removed, we already have version pinning and dependency resolution implemented in package managers. Given each package manager has a slightly different approach, we lose their specific features due to such wrapping. Explicit use of package managers would prevent that. Technically, this is a case of Dependency Injection technique in software design.

Allowing to specify how to build a 3rd party package from a metadata file is a way to hell... I'm strongly against that. If absolutely needed, rdmanifest can be used for this.

@peci1 Could you explain what part of the proposal led to this comment? <env_name>.depends.cmake.repositories key?

[peci1]

@peci1 Could you explain what part of the proposal led to this comment? <env_name>.depends.cmake.repositories key?

Yes, both the cmake, meson and (future) others.

[tfoote]

This proposal targets multi-package ROS projects and aims to streamline build and dependency management for basic and advanced use cases. In the following, we call a multi-package ROS project a ROS package collection and treat it as a single entity to manage.

A ROS package collection is assumed to be developed on a single version-controlled online repository by a single organization. We assume that the // uniquely identifies the collection. The organization has full control over all packages in the collection.

I think that a lot of this capability can be had by using the existing path of forking the rosdistro and leveraging all the existing toolchains. This is a moderately common practice for internal corporate deployments. When you start proposing enabling selecting specific versions of dependencies you are effectively creating your own fork of the distro and will need to take on all the QA and cross compatibility validation. Libraries and programs are no longer guaranteed to be compatible with ones from the original distro.

I would strongly recommend against trying to develop a new concept to capture this and instead look at the existing toolchain and identify what use cases are not working well and look at how to serve them better by extending the existing tools.

[peci1]

I fully agree with Tully. The one thing which comes directly to my mind regarding specifying exact versions is pip. This "package manager" creates nothing like a distro with mutually compatible packages.

[doganulus]

I would strongly recommend against trying to develop a new concept to capture this and instead look at the existing toolchain and identify what use cases are not working well and look at how to serve them better by extending the existing tools.

The proposal does not aim to re-implement the concept of ros workspace, ros distribution, and custom ros-specific tooling. This is a separate path that prioritizes and helps create isolated development environments and containers, which can be used for C++, Python, and Rust applications that use ROS client libraries.

You can have other proposals and features to improve existing tooling; but I do not cover them under this proposal. Your comments about the aforementioned direction and this proposal are always welcome. Even if they are hypothetical comments. Thank you for your contributions.

[doganulus]

Allowing to specify how to build a 3rd party package from a metadata file is a way to hell... I'm strongly against that. If absolutely needed, rdmanifest can be used for this.

@peci1 Could you explain what part of the proposal led to this comment? .depends.cmake.repositories key?

Yes, both the cmake, meson and (future) others.

The build is delegated to cmake or meson or colcon or others but we make it explicitly so we do not lose tool-specific features, which is a major problem when wrapping.

Moreover, the vcstool+colcon is already a common practice in the ROS community. This proposal adds git+cmake or git+meson or others.

Finally I would suggest everyone check CMakePresets.json for a modern approach for build configurations.

[peci1]

@doganulus Could you maybe expand a little about how you imagine the whole thing working? I just can't get rid of the impression that what you're suggesting here is a build script that would normally be written as shell, but transformed to yaml. Why not directly a shell script?

My current understanding of your proposal:

• have a folder where sources are checked out via a combination of vcstool and git (why should git be separated out? vcstool can handle git and it can merge multiple .repos files)
• have a common install (and possibly build) folder
• have a definition file (you suggest yaml, I say it looks like a shell script) that defines how each of the checked out packages should be built - i.e. its flags, the buildtool, and possibly also some presets of the flags

I agree such system would result in a pretty nicely defined environment which could be easily containerized.

It basically looks like a meta-buildtool. Oh wait, that's colcon, isn't it? And if colcon doesn't handle meson or cargo - is there a principial reason why there couldn't be a colcon plugin for them?

I know you're often argumenting with the fact that no other project requires such a large custom buildtool set as ROS does. However, most of the projects out there are kind of centralized at least in the sense of the buildtool used to build everything. ROS offers a wide variety of build types and that requires a kind of meta build tool so that people can still call a single command to build everything. Except for debuild and rpmbuild (and the more exotic ones like Nix, conda etc.), I don't know any other meta build tool ROS could rely on. So, there is colcon. And you're suggesting to build a meta-meta-buildtool... I don't like this direction. I'd incline much more towards keeping colcon as the top-level buildtool and adding support for whatever the community comes with to it.

I'm not (yet) very proficient in the capabilities of colcon/ament, but it seems to me most of the things you propose could become just options of these tools. Maybe it's not too common to commit these buildfiles into a repo, but theoretically, there isn't anything preventing you to do that, or is there?

Maybe it would help a bit to formalize a way to provide a package.xml externally so that the ROS developer doesn't need to fork a package just to add this file?

[doganulus]

Could you maybe expand on how you imagine the whole thing working? I just can't get rid of the impression that what you're suggesting here is a build script that would normally be written as shell, but transformed to yaml. Why not directly a shell script?

Your impression is not wrong, indeed. I envisage that an environment specification in the proposal is directly translatable to a shell script, which is executed to prepare your development environment (probably in a container). Here I also abandon the workspace concept as third-party packages are installed in conventional directories (eg. /opt/<organization>/<collection>/packages) rather than copied into the workspace. So you going only to build your (collection) packages, not others, once the devel environment is built.

The proposal does not dictate a particular implementation, but such a translator-to-shell approach is one of the possible implementations. Once we have well-defined specifications and conventions, I think multiple implementations should be encouraged, which will enrich community development and experiments. However, as exemplified in previous comments, the current design creates a vendor lock-in in tooling and minds. Therefore, achieving plurality in tooling is very important in this proposal.

It may be helpful to mention docker-compose as one of the inspirations for this proposal. You know compose.yml is a declarative format that orchestrates multiple (docker) CLI commands, and YAML fields correspond to Docker CLI flags more-or-less. In my experience, that's more convenient for the user than dealing with long CLI commands and scripts. Therefore, I would call this proposal a CLI orchestrator targeting package managers and build tools to construct development environments.

[arjo129]

It basically looks like a meta-buildtool. Oh wait, that's colcon, isn't it? And if colcon doesn't handle meson or cargo - is there a principial reason why there couldn't be a colcon plugin for them?

Minor knit: Colcon has plugins for cargo. I think the current issue is that there is no way to bloom a rust package. The debian folks have some policy around this (https://wiki.debian.org/Teams/RustPackaging) but I'm not sure how that'll play out with the build farm.

I feel this proposal is too complex. I might as well write a docker file and call it a day. I think before attempting a proposal like this itd be good to have a clear pain point. Perhaps the pain point can be "I can't use rclpy with my favorite python library" and start from there (I've faced this and its kind of annoying -but it may not be within the scope of the ROS 2 project to solve). It makes sense that rosdistro be based on stable libraries as thats the first entry point for users.

[doganulus]

I feel this proposal is too complex.

Dependency management is inherently complex. Hiding it under single rosdep keys does not handle such complexity beyond basic use cases. This proposal is designed to scale from one-file one-environment cases to multi-file multi-environment cases. Those are all necessary in production.

I might as well write a docker file and call it a day. I think before attempting a proposal like this itd be good to have a clear pain point.

The initial pain point is that I am not able to use native tooling when developing ROS-based applications. The current design does not allow me to step outside easily (maybe intentionally, maybe not). Therefore, I want to design a system that does not create vendor lock-in. Therefore, this proposal does not intend to block pure Python and Rust app development and distribution methods. I recommend the use of requirements.txt and cargo.toml as usual. As I stated, I do not aim to support the concept of ROS distribution. Distribute your packages on GitHub, GitLab, Pypi, and Cargo as everyone does.

You can use a shell script, dockerfile, or ansible script to create your environments. This is a declarative specification of what goes inside with extension and overriding semantics. This is not a tool by itself. Akin to package.xml but at collection level as our robotic systems grew more complex in the past decade.

[arjo129]

Therefore, I want to design a system that does not create vendor lock-in.

What is the vendor lock-in we are fighting??

As I stated, I do not aim to support the concept of ROS distribution.

Why bring it up in the ros2 project then? There are tonnes of projects that do this (docker, nix-os) etc.

The initial pain point is that I am not able to use native tooling when developing ROS-based applications.

This seems to be a valid point. Perhaps you are looking at this the wrong direction. Have you taken a look at what the folks at robostacks and pixi.dev are doing.

[peci1]

Therefore, I want to design a system that does not create vendor lock-in.

What is the vendor lock-in we are fighting??

I second this. What is the problem you're trying to solve? Is it so that you can't have a cargo.toml package that uses rclrust and builds directly using cargo? Or something else?

[doganulus]

Therefore, I want to design a system that does not create vendor lock-in.

What is the vendor lock-in we are fighting??

I second this. What is the problem you're trying to solve? Is it so that you can't have a cargo.toml package that uses rclrust and builds directly using cargo? Or something else?

The stated goals are as follows:

1. Collections as a distribution unit for large ROS-based projects.
2. Environment-wise system and source dependency declarations.
3. Well-defined override and extension semantics.
4. Graceful scaling from the basic use case to advanced production ones.

But, of course, any pure Python and Rust ROS applications should be able to distribute their packages using their native methods (Pypi and Cargo), and other ROS projects (including polyglot projects) may declare dependencies on them. Among others, the proposal ensures that multiple package managers can use their own keys directly.

[peci1]

You haven't answered what is the current vendor lock-in. Without knowing that, it's hard to move anywhere (because it seems you're proposing to create another vendor lock-in resolving the original vendor lock-in). And no, creating a "standard" with multiple implementations doesn't mean it's not a vendor lock-in (as long as the standard exists just for ROS).

[doganulus]

You haven't answered what is the current vendor lock-in.

As I have stated, the initial pain point is that I am not able to use native tooling when developing ROS-based applications. No other software library does that to me.

Of course, we will have open standards and conventions. This is not a lock-in by itself. But if we design it in a way that prevents users opt-out, then yes it is a hostile standard for vendor lock-in, especially if the standard targets our own products.

This proposal can be equally useful for Zenoh applications. Just there is not any large project on Zenoh as far as I know. And they do not dictate any tooling and can perform multi-language pub-sub already.

[peci1]

I still don't see it (but my world is mostly in CMake, so maybe I'm missing something specific to Py or Rust packages).

If I have a binary distribution of a C++ ROS package, I can create a dependent package that doesn't need to use catkin/colcon, but is plain cmake, and is built by the classical mkdircd build && cmake .. && make && make install (when paths are correctly set up).

If I have a source-distributed C++ CMake ROS package, I can create a dependent package as well, because even the dependent package can be built without colcon using the standard CMake way.

The "only thing" that catkin/colcon brings is correct ordering the package builds. If you handle that in a different way, you don't need catkin/colcon.

So, what does "I am not able to use native tooling when developing ROS-based applications" exactly mean?