Fix broken/flaky docker image builds

[fhennig]

Related PRs:

• Docker build cache: https://github.com/stackabletech/docker-images/pull/735
• Split CI per product: https://github.com/stackabletech/docker-images/pull/733

Split CI per Product

To be able to run individual workflow for each product, we introduced various changes:

• Split up conf.py files. Now each product keeps track of the versions inside its own folder. This requires an up2date version (0.0.10) of image-tools. See https://github.com/stackabletech/docker-images/pull/732
• Pull out common functionality into local actions in .github/actions. There are now four actions:
  • shard: This dynamically generates a list of versions which is then used to define the matrix used in the build step.
  • build: This action runs the appropriate bake command to build a product in a specific version.
  • publish-image: This publishes the images and SBOMs to both Nexus and Harbor.
  • publish-manifest: This publishes the manifest of a product image for one version for both amd64 and arm64.
• We don't use shard count and shard index anymore. Instead defined versions are used to populate the matrix.
• We adjust trigger paths to only build images we need (or depend on other changes).
• We also run the workflows every second day at 01:00 in the night.

Distributed Docker Build Cache

The bake command from the image-tools-package package can configure docker build cache backends. The only economically viable backend for now is the registry backend.

Unfortunately this backend is broken for multi-stage images. See the discussion here.

This means we do not use full potential of the docker cache because only the intermediate images (and their layers) can be reused from cache, but not the layers of the final image. This is unfortunate because the final images are usually also the most expensive.

Proposal: spike splitting up multi-stage images into multiple single-stage images and test the caching impact on the build performance.

Meeting notes 1.7.2024:

Participants: on-prem

• Split up conf.py in multiple files per product
• manually change something in the dependants and use path attributes when triggerring GH actions
• Look into using a distributed build cache backend: https://docs.docker.com/build/cache/backends/
  • look into GH container registry
• Have an automatically scheduled job that rebuilds everything (e.g. every second night), which does not use the cache
• Don't use caching for release builds
• keep the existing workflow to rebuild everything and add a new one to build individual images
• If the run-times on main branch are good enough (whatever that is), build the affected images in PRs
• related ticket to build Java products outside of docker: TODO: add link here

Meeting notes 20.6.2024:

Participants: @lfrancke , @dervoeti , @Maleware , @siegfriedweber , @razvan

Notes:

• We want to rebuild the manifest list of a product version as soon as possible
• We want to replace the existing workflow for dev images to only build images when necessary (when the corresponding paths have been changed).
• changes to conf.py are hard to identify
• adjust the numbers of shards per product
• fix the current issue where we assume that one runner builds only one image (bake-tags file can contain multiple tags)
• consder rebuilding depenants like hbase also when hdfs has changes

Not part of this issue, but related:

• Use Nexus as Maven pass through cache

Look into potential helpers:

• https://docs.github.com/en/actions/using-workflows/reusing-workflows

As discussed in the retro

to be refined.

### Tasks
- [ ] https://github.com/stackabletech/docker-images/pull/725

[NickLarsenNZ]

FYI, reusing workflows doesn't work too well with matrices (if you need to pass specific stuff between jobs). Not impossible, but will need a creative solution.

[fhennig]

We want to replace the existing workflow for dev images to only build images when necessary (when the corresponding paths have been changed).

I have investigated this before, and it is not straightforward to have mandatory checks that only need to run for changes in certain repositories. If a check is only running on certain changes, but it is also mandatory for the check to complete, the PR will be in an un-mergable state. Online a few people developed workarounds, but it's not pretty.

Just wanted to share this since I already looked into it before.