search

bioconda / bioconda-utils

Utilities for building and managing bioconda recipes
MIT License
96 stars 133 forks source link

feat: simplify unify containers #959

Open daler opened 6 months ago

daler commented 6 months ago

This takes the ideas from the code review on #953 and simplifies by refactoring how everything is configured. See https://github.com/bioconda/bioconda-utils/pull/953#issuecomment-1958104290 for the specific proposed items, which are addressed here.

The only major difference from those proposed items is storing the conda/mamba versions, which are not clearly defined in requirements. For example, I think mamba comes in from the base image (quay.io/condaforge/linux-anvil-cos7-x86_64). So I kept the same strategy as before, where the create-env depends on the build-env and inspects the conda/mamba versions used there.

Edits 2024-08-10

Here's an overview of the changes...

This is strictly for the base images. They are used for creating images for packages, and once this is merged they will will be pushed to the same quay.io tags as existing base images. But this PR does not deal with creating package images.

(if the nomenclature of images is confusing, see dockerfile inventory for a guide)

The package building has been unified to use a single build.sh script which takes an image directory as its only argument. That image directory should have prepare.sh, Dockerfile, and Dockerfile.test. Each <IMAGE DIR>/prepare.sh sources versions.sh which is the common place to set versions and tags. It also provides some helper functions. You can read about all of this in images/README.md.

The new build-images.yml GitHub workflow handles the interdependencies of the image building and testing. It uses GitHub Container Registry as a temporary storage space for images. After testing, tagged images are pushed to quay.io.

The main advantage of this overhaul is to factor out all the commonalities across building images, which otherwise have grown in different directions over the years. This provides just one unified system that maintainers need to understand. The reason it is in this repo rather than bioconda-containers is that we have strong dependencies on the version of bioconda-utils, so ideally changes to bioconda-utils should also test that corresponding base images work correctly.

NOTE: this does NOT yet replace the bioconda bot image so bioconda-containers is still needed for that.

daler commented 1 month ago

Some notes on the current status of this:

I would like to be able to test the just-built images (in the case that they were rebuilt in a PR) but also test the existing images on quay.io (in the case that only bioconda-utils has been changed).

This requires a clean way of specifying which images to use, but this is currently complicated. So the next stage here is cleaning up how images are specified throughout the code base. To avoid needing to simultaneously update mulled-build source, the strategy here should be to match what mulled-build looks for and uses.

Specifically in these lines of mulled_build.py, we have:

daler commented 1 month ago

I noticed that the build-env container is installing a version of conda that is not specified in bioconda-utils requirements. Can't quite figure out why. Despite requirements saying conda=24.5.*", it's installing23.7`...and it's taking almost an hour. Going to revert f311612 which removed mamba...