Pegging workspace to SHAs / tags / branches

[agalbachicar]

In favor of a clearer development life cycle and towards open sourcing / sharing the workspace to partners we need to think again our development workflow. There are a few use cases that would need to be covered by this change in development:

• Recurrent set of SHAs / tags / branches to checkout each package to and have a stable version.
• Ability to bisect the workspace to understand where a bug was introduced.
• Release automation of the solution

Out of the scope for now, but desirable at some point in the future:

• Semantic versioning
• Release automation of the solution

Victory condition

A new development workflow is proposed that easily allows to:

• checkout to each package to a coherent version,
• build the workspace,
• use the workspace,
• generate or get a workspace tarball

[liangfok]

As I work on updating our downstream project to the latest version of Maliput/Dragway/Multilane/Malidrve, I just realized that having sets of SHAs that are known to work together would be very helpful. By having these sets of SHAs, I could incrementally bump our downstream project such that any failure is less likely due to the combination of SHAs I happen to choose.

This previously did not occur to me because our downstream project has never been this far behind (it's stuck on SHAs from early Novmber 2020), meaning I would simply update our downstream project to the latest versions of everything. While I could still do that, I am somewhat hesitant given the massive (but good) changes that have occurred since November of last year.

[stonier]

Let's do it then.

Proposal

Repos Files

• Add a maliput-stable.repos to this repository.

Don't worry about ignition-stable.repos or delphyne-stable.repos until we feel need to test the stable release separately (it's too volatile anyway). Likewise, don't worry about customising it to just the tri-used repos (the less that is custom, the better)

Branching & Tagging

• Bump a minor version in each package.xmls in each repository
• Tag it according to the major.minor.patch version
• Create a branch release/major.minor.x , e.g. release/1.1.x for each repository (use this in the .repos files)
• Clean up other branches (we have alot of old branches lying around)

This avoids churn on the repos files constant even as patches come in.

There is some fine print here - it assumes that all packages in a repository have the same version number for it to really make sense, which ends up not making sense if your packages are evolving at different rates, or your packages don't follow this convention. I think we can be ok with this pragmatism though, we don't have large multi-package repositories.

Alternatives

• When repositories are multi-package, include a meta-package that depends on the others. Version this separately and use that for tagging and branching purposes.
• Use nonsense names for repo versioning and tagging, e.g. pink-elephants-0, pink-elephants-1 (first patch on pink-elephants), green-flamingoes-0.
• Use dates for repo versioning and tagging, e.g. tag 20210305 → more churn on branches this way though. You could add the -N suffix for tags and just the initial creation date for the branch though.

[liangfok]

Sounds good! From a brief meeting with @agalbachicar and @francocipollone this morning, I agreed that having named releases rather than sets of SHAs is preferable. I just happen to fixate on SHAs since that is what our downstream project uses when pulling in these Maliput-related repos.

Question: What if only one of the N repos contains changes that warrant a new release? Would that trigger new releases for all of the other N-1 repos even though they did not have significant changes? The example I can think of is the removal of proj4 from Malidrive (https://github.com/ToyotaResearchInstitute/malidrive/commit/792ccf883a6ac93b68cd7e9e99ffa069eb8d4ea1). That only impacted Malidrive, not all of the other repos.

[liangfok]

I suppose there is no need to keep the various repos in sync w.r.t. releases since there is an overall maliput-stable.repos file that ties everything together. For my task of updating the downstream project, I would just refer to the evolution of this file.

[scpeters]

I made an example of exporting a .repos file from a GitHub workflow in https://github.com/ToyotaResearchInstitute/maliput_infrastructure/commit/f06813569387e3634a3ea6382e98e512fdde42b3 (branch scpeters/archive_exact_repos_file)

• https://github.com/ToyotaResearchInstitute/maliput_infrastructure/actions/runs/768807403

I can make a pull request for this if people are interested

[agalbachicar]

I think we can start from there as a step towards the completion of this task. Thanks @scpeters !

[agalbachicar]

I would like to ask a few questions that will help to make proper docs and then try to automate this process.

• Add a maliput-stable.repos to this repository.

  • @stonier did you mean https://github.com/ToyotaResearchInstitute/repos_index ?

• Do we start with 0.1.0 or 1.0.0 in all repos?

• it assumes that all packages in a repository have the same version number for it to really make sense, which ends up not making sense if your packages are evolving at different rates, or your packages don't follow this convention. I think we can be ok with this pragmatism though, we don't have large multi-package repositories.

  • Agreed! For the record, we just have maliput repository which contains: maliput and maliput_py packages. And then delphyne_gui repository which contains: delphyne_demos and delphyne_gui packages. We could have another two repositories for them if you think it'll standardize the workspace and simplify the development lifecycle. Impact in CI should not be that big if we are careful with tests.

• What do you think would be the most appropriate releasing scheme? Based on features or based on a fixed period (i.e. 2 weeks, 1 month, etc.)? I am interested in knowing what would help you when integrating downstream code @andrewbest-tri @liangfok @stonier

[agalbachicar]

I've created https://github.com/ToyotaResearchInstitute/maliput-documentation/pull/30 to discuss the details of the releasing process.

[agalbachicar]

Next steps:

• [x] Add changelogs to all packages following ROS2 template. -> @gbiggs
• [x] Split delphyne-gui into delphyne_gui and delphyne_demos. -> @agalbachicar
  • [x] delphyne_demos package must be migrated to delphyne_demos repository. https://github.com/ToyotaResearchInstitute/delphyne-gui/pull/397 https://github.com/ToyotaResearchInstitute/delphyne_demos/pull/1
  • [x] Fix maliput_infrastructure CI. https://github.com/ToyotaResearchInstitute/maliput_infrastructure/pull/193
  • [x] Fix delphyne.repos in repos_index. https://github.com/ToyotaResearchInstitute/repos_index/pull/3
• [x] Split maliput into maliput and maliput_py. -> @agalbachicar
  • [x] maliput_py package must be migrated to maliput_py repository. https://github.com/ToyotaResearchInstitute/maliput/pull/408 https://github.com/ToyotaResearchInstitute/maliput_py/pull/1
  • [x] Fix dependencies in downstream packages like maliput_dragway and others. https://github.com/ToyotaResearchInstitute/maliput-dragway/pull/41 https://github.com/ToyotaResearchInstitute/delphyne/pull/773 https://github.com/ToyotaResearchInstitute/delphyne-gui/pull/394
  • [x] Fix maliput_infrastructure CI. https://github.com/ToyotaResearchInstitute/maliput_infrastructure/pull/190
  • [x] Fix maliput.repos in repos-index. https://github.com/ToyotaResearchInstitute/repos_index/pull/2
• [x] Use main instead of master everywhere https://github.com/ToyotaResearchInstitute/maliput_infrastructure/issues/191
• [x] Use underscores everywhere
  • [x] Rename maliput-integration, maliput-dragway, maliput-multilane, maliput-integration-tests, maliput-documentation and repositories and packages to use _ instead of -.
  • [x] Fix maliput_infrastructure CI
  • [x] Fix repos files in repos_index #191
• [ ] First release
  • [ ] Define initial release number in all packages. Mature packages should adopt version 1.0.0. Immature packages should use 0.1.0.
  • ament_cmake_doxygen -> 1.0.0
  • maliput -> 1.0.0
  • maliput_py -> 0.1.0
  • maliput_dragway -> 1.0.0
  • maliput_multilane -> 1.0.0
  • maliput_malidrive -> 0.1.0
  • maliput_integration -> 1.0.0
  • maliput_integration_tests -> 1.0.0
  • malidrive -> 1.0.0
  • delphyne -> 1.0.0
  • delphyne_demos -> 0.1.0
  • delphyne_gui -> 0.1.0
  • drake_vendor -> 1.0.0
  • maliput_documentation -> 0.1.0
  • [ ] Prune dead branches in all repositories.
  • [ ] Define the name of the release.
  • [ ] Follow the steps to make the first named release.

[agalbachicar]

@stonier / @andrewbest-tri / @liangfok would you create two new repositories?

• delphyne_demos
• maliput_py

[liangfok]

Done.

https://github.com/ToyotaResearchInstitute/maliput_py https://github.com/ToyotaResearchInstitute/delphyne_demos

Do I also need to set the MALIPUT_TOKEN secret?

[agalbachicar]

Yes, please!

[liangfok]

Done!

[scpeters]

• Define initial release number in all packages. Mature packages should adopt version 1.0.0. Immature packages should use 0.1.0.

I recommend that we follow semantic versioning for these releases, such that incompatible changes to the API require an increment to the major version once we have reached 1.0. We can also decide on a policy for ABI changes, but that is not critical until we are providing binary packages

There is a tool we use in the ignition buildfarm to check for API and ABI breakage:

• https://github.com/lvc/abi-compliance-checker
• https://github.com/ignition-tooling/release-tools/blob/c819f8682070ee88a9ac80b0983205c5cda5141f/jenkins-scripts/docker/lib/generic-abi-base.bash#L109
• example report: https://build.osrfoundation.org/job/ignition_plugin-abichecker-any_to_any-ubuntu_auto-amd64/91/API_5fABI_20report/