Our image building could be much more efficient - we don't preserve layers between builds, even when the contents of a layer could be preserved between builds (e.g. we have 23(?)?? images which separately install micromamba)
Something fun would be reconsidering our images, separating static content (OS, compilers, system packages etc.), semi-static content (versioned python installations? installation of our own hail fork/version), and variable content (our internal software libraries, 3rd party tools we expect to update regularly, versions linked to individual repository hashes) and come up with a more interactive build plan:
Clean up our builds into a range of ingredient layers (so as to improve the number of common layers across our infra)
When triggering the build process, we check which images are due to be rebuilt (similar to the current matrix generation)
SOMEHOW we parse the build files and create a graph of the layers which need updating
If a Dockerfile to be built uses a base layer which itself needs to be built, build from root to leaf in order (on the same CI worker, and push all results)
If a Dockerfile needs to be built, but the base layer it depends on is unchanged, pull the base layer, then build on it
For context, I do not have a great idea of how to solve this issue. Or if it truly is an issue.
Vaguely related: https://github.com/populationgenomics/analysis-runner/pull/682
Our image building could be much more efficient - we don't preserve layers between builds, even when the contents of a layer could be preserved between builds (e.g. we have 23(?)?? images which separately install micromamba)
Something fun would be reconsidering our images, separating static content (OS, compilers, system packages etc.), semi-static content (versioned python installations? installation of our own hail fork/version), and variable content (our internal software libraries, 3rd party tools we expect to update regularly, versions linked to individual repository hashes) and come up with a more interactive build plan:
For inspiration: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/docker/build_docker.py
What we hope to gain from this: