Support for Package Sources

[cottsay]

Fedora recently introduced a system they call 'Modularity' which is intended to address the too fast/too slow problem with packages.

This problem is a major pain point for running ROS on Fedora, because ROS system dependencies typically follow whatever versions are in the targeted Ubuntu releases at the time. It is very common for Fedora's package version to be different (and incompatible) with the versions used by Ubuntu, and therefore targeted by a particular ROS release. A great example is Gazebo, where Melodic targets Gazebo 9 and Fedora currently packages Gazebo 8.

Please feel free to read the Fedora Modular docs [1], but to summarize the parts we care about here, a module (e.g. 'nodejs') has one or more streams (e.g. '8' or '12.x', or 'lts') and those streams contain differing versions of packages. When the module is enabled, packages from that module and selected stream are made available on the system.

There doesn't seem to be a precedent for this type of behavior in rosdep, but I think we could probably simplify the problem by thinking of modules as a package source of sorts. I would suggest two possible ways to implement this:

1. Treat package sources as packages themselves, and either:
   1. The required package can be installed implicitly as part of enabling the source. I went ahead and made a sample implementation of this (see #642). This might look like:

      nodejs8:
        fedora:
          dnf-module: [nodejs:8]

   2. The package could leverage the existing depends support in rosdep, and the package and the source would then have separate rosdep keys. The module would only be enabled by the second key - no packages would be installed.

      nodejs8:
        fedora:
          packages: [nodejs]
          depends: [nodejs8-module]
      nodejs8-module:
        fedora:
          dnf-module: [nodejs:8] 

2. A new concept (something like PackageSourceInstaller) could be added. This might look something like:

   nodejs8:
     fedora:
       packages: [nodejs]
       sources:
         dnf-module: [nodejs:8]

I believe that (2) is the most clear, and could also be leveraged for many other opportunities, such as 3rd party repositories or Fedora COPR [2].

I'm hoping for feedback on whether (2) would be desirable, or if I should just work around the problem using existing concepts as demonstrated by (1).

Thanks!

--scott

@richmattes FYI

[1] https://docs.fedoraproject.org/en-US/modularity/ [2] https://developer.fedoraproject.org/deployment/copr/about.html

[richmattes]

I really like this for being able to pull other package sources that are built into operating systems' package managers. I agree that (2) looks the nicest, and I can see it scaling to other sources like ppa, copr, etc. And I wonder if it could also be used to support secondary package managers like pip, or if that would be out of scope.

For Fedora specifically, do you envision e.g. a Gazebo module with versions 8 and 9, which also includes the ignition and sdformat dependency graph, that would be enabled based on which ros distro you were using? If so, that would definitely alleviate a lot of the Ubuntu/Fedora incompatibilities that pop up as a barrier to maintaining Fedora packages in parallel with Ubuntu, as long as the required package sources are reliably available.

[NikolausDemmel]

We had some thoughts on how to support custom PPAs in xylem (which at some point was started as a ROS agnostic successor to rosdep, but due to lack of time and maybe need never got far enough). Those were just our ideas at the time, but the general concept is similar to (2), allowing additional package-manager specific options: https://xylem.readthedocs.io/en/latest/design.html#installer-options

[cottsay]

Besides Fedora Modular, another motivation to solve this issue is the Red Hat Software Collections (RHSCL) concept [1], which is necessary to get the tools required to build ROS 2 (and probably a fair sized chunk of ROS 1 as well).

There is a conversation that needs to happen regarding which sources are appropriate for inclusion in the central rosdep database, but I'd like to talk about that in another setting. Right now, we need to get the framework laid out.

[1] https://developers.redhat.com/products/softwarecollections/overview/

[tfoote]

In general we have a specific set of sources for which the rosdep keys are valid. If you are using a specific set of rosdeps they have a set of sources which are expected to already be on your system. For example the main rosdep database expects Ubuntu Universe + ROS sources. There is a gazebo set of rosdeps that expects the OSRF repositories.

In general I think that it makes sense to isolate things that way such that a particular set of rosdep keys requires specific sources. Then when you add that rosdep source to your rosdep database you also enable the database/repository etc that the set of keys needs.

It would be good to consider dropping a marker into the rosdep sources that could test/assert that the source is present. But I would strongly suggest that we not have a way to automatically add sources, nor that we allow sources per rosdep key. The first is a major security risk, and the second is a maintenance burden.

[cottsay]

Thanks for giving your thoughts, @tfoote. I think you brought up some really good points. This made me come to the realization that switching or enabling a Fedora Module could possibly reinstall a package, which could result in any number of packages being uninstalled, which is something I would never want rosdep to do automatically. I don't believe that is the case for RHSCL, which I think is a better example of an in-house channel that we might want to enable automatically, but I can see how this might be a slippery slope.

I have some ideas for how we could make this a little more clean for Fedora Modular, but I'll need to do some testing.

It would be good to consider dropping a marker into the rosdep sources that could test/assert that the source is present. Would we add this marker per-key? Or globally, sort of like a "readiness check"?

Do you think there would be value in adding a "hint" that gives a URL or brief instructions on where the package might be found? Something like:

nodejs8:
  fedora:
    hint: "Provided by the DNF module 'nodejs:8': To enable, run 'dnf module enable nodejs:8'"

[tfoote]

I like the idea of a hint, but I'd rather not see it per key, but per source that might be required. Thi ssort of hint is what the user would be required to do to add a trusted source.

For something like the dnf module enable, (which i'm not familiar with so I might be off base) that sounds like it's just enabling something that's already on your system or otherwise installable so that should be automatible. The dnf keys could be extended to support both installing packages and enabling modules with an entry like {package:[packages], modules: [modules]}