Support specifying custom dependencies (not via rosdep) and/or automatic dependency resolution

[AAlon]

Enhancement proposal.

Option 1: dependency manifest

Would look similar to the format of the base.yaml file but with ROS distro mapping, for example:

dependency_manifest.yaml

kinetic:
   aws_common:
       ubuntu: [ros-kinetic-aws-common]
melodic:
  aws_common:
      ubuntu: [ros-melodic-aws-common]
      debian:
          stretch: [ros-melodic-aws-common]

bloom-release <key> --track .. --rosdistro .. --custom-dependencies dependency_manifest.yaml

I suggest rosdep rules would still take precedent, so if we have two conflicting values - whatever's in distribution.yaml / base.yaml would apply.

Advantages:

• Support multiple distros, hence would make it easier to automate the release
• No need to modify system wide dependency configuration for releasing packages
• Explicit - less prone to human error

Disadvantages:

• Still need to create & maintain this file

Option 2: automatic inference

We could support a more implicit approach of guessing the dependency key and assuming it will exist. bloom-release <key> --track .. --rosdistro .. --guess-missing-dependencies

So if a package depends on some-ros-package which hasn't been released yet (i.e. it's not in distribution.yaml nor in base.yaml), bloom would infer the key to be ros-<distro>-some-ros-package.

If run in interactive mode, bloom would ask the user to confirm its assumed mappings.

I'm leaning towards Option 2, since bloom itself already knows: 1) what would be the resolved name of a dependency (aka ros-<distro>-pkg) 2) which OS & versions are relevant for each ROS distro

So I don't see why would the user need to specify it explicitly (for arbitrary overrides folks could still use rosdep files).

Related to https://github.com/ros/rosdistro/pull/20670

[tfoote]

In general I don't think that we want to add support for injecting dependency resolution into bloom when making releases. The reason that bloom uses the rosdistro and rosdep keys is that these are the same as what the buildfarm will support. If bloom can resolve the dependencies then the buildfarm will be able to resolve the dependencies. If we provide mechanisms for injecting dependency names outside of that mechanism then the odds of a user "fixing" a name resolution to make bloom happy goes way up, but that often will just fail later on the buildfarm.

The current approach for doing this is to add a custom rosdep rule before running bloom. There are two known use cases for doing this. The first is if you are running a custom version of the buildfarm or other private instance with additional dependencies maintained in a custom repository that is expected to be made available on your system. The second use case for this is if you are doing a first time release of new packages and you want to do release multiple layers of dependencies simultaneously.

For the first use case the additional rosdep dependencies is effectively the right thing to do. In the second case it's more of a hack that's manually pre-resolving the dependencies of releases that are expected to have been published (but not currently published) prior to the current release being published. This is inherently fragile and requires very active management by the release manager. The simplest solution is to simply make the releases in dependency order. If there's 4 layers of dependencies it will require 4 release cycles for the first release into a new distro. After they've been released for the first time into the distro they can all be resolved and it's no longer an issue. If that's really not possible the release manager must have a list of resolutions expected to inject. As I mentioned above I'm not in favor of adding a new injection mechanism. To simplify building that list a simple script could be made to list packages and output the rosdep.yaml format with the packages names templated into the debian package names for a specific rosdistro that can then be introduced into the rosdep resolutions. And then since there's no need for those ongoing rules to be available they should be immediately removed.

If the multilayered release is being automated with a script the injection and removal of the additional rosdep sources could be relatively easily automated as well.

[AAlon]

If we provide mechanisms for injecting dependency names outside of that mechanism then the odds of a user "fixing" a name resolution to make bloom happy goes way up, but that often will just fail later on the buildfarm.

The thing is, users already have to fix the name resolution via a custom rosdep rule to make bloom happy... so it's less a question of whether to provide the ability to inject names, more about how to provide it better.

In the second case it's more of a hack that's manually pre-resolving the dependencies of releases that are expected to have been published (but not currently published) prior to the current release being published. This is inherently fragile and requires very active management by the release manager. The simplest solution is to simply make the releases in dependency order. If there's 4 layers of dependencies it will require 4 release cycles for the first release into a new distro. After they've been released for the first time into the distro they can all be resolved and it's no longer an issue. If that's really not possible the release manager must have a list of resolutions expected to inject. As I mentioned above I'm not in favor of adding a new injection mechanism.

If the multilayered release is being automated with a script the injection and removal of the additional rosdep sources could be relatively easily automated as well.

A handful of developers could be responsible for maintaining 30 packages with 6 layers of dependencies; Releases for ROS2 may happen in 6 months intervals. Having to release in layers every time would be too much overhead. We need to think about streamlining, simplifying and automating the release process as much as possible.

To simplify building that list a simple script could be made to list packages and output the rosdep.yaml format with the packages names templated into the debian package names for a specific rosdistro that can then be introduced into the rosdep resolutions. And then since there's no need for those ongoing rules to be available they should be immediately removed.

That basically means that every developer (or team) would need to solve this problem for themselves. Why not go with the automatic inference option then, so we solve it once and for all? (at least for the vast majority of users). We could present an additional confirmation prompt to make sure that users are aware of what the --guess-missing-dependencies is, and that they're expected to release missing dependencies in tandem.

[nuclearsandwich]

The thing is, users already have to fix the name resolution via a custom rosdep rule to make bloom happy...

From Tully's comments, a package maintainers targeting the official build farm should not fix dependency resolution issues locally as they are just pushing the breakage further down the line.

For package maintainers targeting some other rosdistro I think that making workarounds easier optimizes in the wrong direction and I would rather recommend that new contributions focus on improving the experience of maintaining and utilizing alternate rosdistro and rosdep databases.

Having to release in layers every time would be too much overhead. We need to think about streamlining, simplifying and automating the release process as much as possible.

As the release engineer who has been primarily responsible for bootstrapping ROS 2 releases since we first started making .deb packages available, I hear you. There's a not-insignificant amount of labor involved with bringing up a ROS distro. I agree that streamlining the release process is desired even with a relaxed release cadence. But if the intent of assuming package resolution is to parallelize releases low-level, that sacrifices the automation goal of failing fast and early as we'll have automated a bunch of releases before learning that we made an invalid assumption.

I would rather we streamline by replacing non-LTS releases with a rolling release modeled on Debian Unstable (as was previously discussed) which would change the bootstrap phase for a new stable release from a from-scratch affair to snapshotting the rolling release. I expect that the improvements made to support this would also make it easier for organizations maintaining forked rosdistros to similarly bootstrap theirs from the official rosdistro.