search

gazebo-tooling / release-tools

8 stars 9 forks source link

Handling dependencies across different tools for the different gz software and versions #190

Open j-rivero opened 4 years ago

j-rivero commented 4 years ago

The software dependencies for our packages are mainly coded for building in CMakeList.txt. When creating building and CI environments the code is hosted in different places:

First feeling shared with @mjcarroll is that rosdep would be a bit too much and to get full support of it we would probably need to implement part of ROS specific stuff like package.xml.

j-rivero commented 4 years ago

Example of places to update when bumping a major release in ignition:

chapulina commented 2 years ago

I don't think anyone would disagree that we should have a standard way of defining the dependencies for each version of each package. I believe there are a couple of axes to be discussed:

Existing approaches

declarative vs imperative

I believe we should get rid of any imperative approaches. dependencies_archive.sh is tough to debug because a developer must follow the entire control flow to understand what dependencies came out at the end. It's also confusing for newcomers when they're looking at CI build logs and the entire dependencies archive is printed every time - what does that mean, what is actually meant to be installed for this build?

Proposal: declarative

distributed vs centralized

I believe we've had different degrees of success with our existing distributed and centralized approaches. I'll speak to the advantages of a distributed approach because that's what I've personally seen the best results with.

Development perspective

I believe the experience with packages.apt has been largely successful for developers. Developers introducing a new dependency can make that change as part of their source code changes and get immediate results from CI. No need to open a separate branch on a different repository, then have to manually configure and trigger jobs, or use features like ci_matching_branch.

In that situation, the centralized approach only holds back developers who are working on source code changes and don't care about how the infrastructure works. The distributed approach empowers them to iterate more quickly.

Maintenance perspective

Changing the dependencies of Ignition libraries, especially Ignition dependencies, currently involves a lot of work. The specific tasks are listed on ISSUE_TEMPLATE/bump_libraries.md, and one can look at issues like https://github.com/ignition-tooling/release-tools/issues/574 to see the large number of pull requests that are necessary to update dependencies. That process has various problems, such as:

I'd love to simplify that, and I believe that a distributed approach would help here, because our source code is distributed. We can't get rid of a PR to the source code. Then why not leverage that PR to also update as much infrastructure as possible? Imagine if we could configure colcon dependencies, macOS dependencies, release dependencies, enable nightlies... All with a single PR? That can be iterated faster, is less error-prone, it's easier for a newcomer to perform, it's easier to be kept in sync... To me, that's the dream.

Proposal: distribute all the things!

scpeters commented 2 years ago

I don't think anyone would disagree that we should have a standard way of defining the dependencies for each version of each package. I believe there are a couple of axes to be discussed:

  • declarative vs imperative
  • distributed vs centralized

Existing approaches

declarative vs imperative

I believe we should get rid of any imperative approaches. dependencies_archive.sh is tough to debug because a developer must follow the entire control flow to understand what dependencies came out at the end. It's also confusing for newcomers when they're looking at CI build logs and the entire dependencies archive is printed every time - what does that mean, what is actually meant to be installed for this build?

Proposal: declarative

distributed vs centralized

I believe we've had different degrees of success with our existing distributed and centralized approaches. I'll speak to the advantages of a distributed approach because that's what I've personally seen the best results with.

Development perspective

I believe the experience with packages.apt has been largely successful for developers. Developers introducing a new dependency can make that change as part of their source code changes and get immediate results from CI. No need to open a separate branch on a different repository, then have to manually configure and trigger jobs, or use features like ci_matching_branch.

In that situation, the centralized approach only holds back developers who are working on source code changes and don't care about how the infrastructure works. The distributed approach empowers them to iterate more quickly.

Maintenance perspective

Changing the dependencies of Ignition libraries, especially Ignition dependencies, currently involves a lot of work. The specific tasks are listed on ISSUE_TEMPLATE/bump_libraries.md, and one can look at issues like #574 to see the large number of pull requests that are necessary to update dependencies. That process has various problems, such as:

  • Lots of PRs need to be open

    • they're so many and so particularly different that we had to create a script to help with that
    • it was so hard to trigger CI for all these branches that we had to implement the ci_matching_branch feature
  • Then all PRs need to be merged in sync

I'd love to simplify that, and I believe that a distributed approach would help here, because our source code is distributed. We can't get rid of a PR to the source code. Then why not leverage that PR to also update as much infrastructure as possible? Imagine if we could configure colcon dependencies, macOS dependencies, release dependencies, enable nightlies... All with a single PR? That can be iterated faster, is less error-prone, it's easier for a newcomer to perform, it's easier to be kept in sync... To me, that's the dream.

Proposal: distribute all the things!

I don't have a problem with some centralization if the data is simple enough. I think gzdev and gazebodistro are pretty useful, but rosdistro is huge and scary to me because of how much it stores. Storing dependency names for different major versions of software on different versions of an operating system feels complex enough for me that it's much better to distribute that data in the packages.apt files.

A caveat: if we stored all our dependency information in package.xml and then auto-generated the content of gazebodistro or even the homebrew-simulation dependencies, I think that would be nice (similar to how bloom autogenerates the -release debian metadata).

So I would take it on a case-by-case basis and not throw out any of our tools just because they are centralized. But I definitely prefer packages.apt files over the _DEPENDENCIES variables in dependencies_archive.sh.

j-rivero commented 2 years ago

The process of using de-centralized packages.apt is ongoing: #705 #707 #704

azeey commented 4 months ago

We now have package.xml files in Harmonic packages, so we can move toward using rosdep for dependency installation.