Making releases based on specific dependency branches is complicated

[jimklimov]

Zproject and its library of known_projects.xml really helps get a multi-component project including its git/rpm/deb/... dependencies off the ground... as long as you are following "master" (or equivalent default) git branches of the projects you are using, and in case of known projects - take them from the common upstream repository.

Due to peculiarities of our own project's release and maintenance strategy, this approach did not work our of the box - but luckily, we can override project.xml as it is being parsed by adding <use tag contents to inject the needed repository (for forked upstreams) and "release" branch name. We can even nest the use-in-use to ensure(*) that prerequisites get built on e.g. travis in correct order, when building from scratch and without packages.

• Nesting has caveats, sometimes top-level "use" tag order matters. This is a bug likely, but separate from this issue.

With 50+ (and growing) components comprising our "distributed monolith" product codebase, it becomes quite tedious to branch and update all the "release" attributes and regenerate zproject's to take into account that they need to checkout that bolted-down source version whenever needed. The overall set of components and their interdependencies also evolve, so preparing a release was also our last chance to make sure every component's project.xml trees made sense on the given month. Painfully, it took a couple of man-weeks to branch in order (and sanity-check) the tree of components involved in a release.

We theorized about some solutions, and it boiled down to a few possibilities, most with some downsides:

• fork zproject to have our own known_projects.xml pointing to our own forks - suffers from using also a correct branch of zproject to also point to correct "releases". And maintenance hell for keeping yet another fork up to date.
• extend upstream zproject for using site-custom known_projects.xml if available (and/or load one like we load licence.xml), as an override to the upstream one... this does make sense, may fit many use-cases, but requires that developers doing the release also have the correct variant of the override XML installed.
• Generate the (includable?) cumbersome tree of dependencies for a project as part of its gsl zproject run, by walking recursively the projects it refers to (if they have a project.xml) - lots of git traffic during project generation and/or needed smartness about already available local workspaces to avoid a new requirement to be online to regenerate a zproject.
• Walk the other projects as part of ci_build.sh - at least it makes most sense, because it is generally used in Travis CI that already is network-capable. Downside is that for prerequisites, tools, etc. that may be needed to build that prerequisite and available via packaging (listed into .travis.yml of that respective project), it is too late for the currently building project to install some, especially if running as unprivileged account - so this approach would likely not fit projects that have very disparate dependencies and tool stacks. It may be however worked around by custom CONFIG_OPTS for the dependency to not require as much as it would for a production package. It also is not a problem for prerequisites available via RPM/DEB/... packaging that already properly refer to further dependencies of their own.

Our team members progressed in the last listed direction, that we'd PR shortly.

Since it is a complex problem with no apparent perfect solution, I separated the problem description from the PR so this can be discussed and maybe progressed independently of that solution.

[jimklimov]

CC @perrettecl @boricj @stark-dev

[boricj]

It's been a while since I've worked on that, but one initial design choice to note is that the recursive ci_dependencies.sh mechanism is backwards-compatible with non-upgraded repositories, as long as all of their recursive dependencies are specified in the topmost project.xml file. This allowed a gradual and seamless roll-out within the 42ity project without regressions nor breakages.