This repository contains Dockerfiles for CI builds for any project that might be interested. The image builds are automated and self-updating, and process described more below in detail. You can see the listing of updated builds at:
⭐️ RSE-OPS DOCKER BUILDS TABLE ⭐️
We use a tool called uptodate run as a GitHub Action that makes it easy to:
For the last two (dockerhierarchy
and dockerbuild
for creating new Dockerfiles and running Docker matrix builds, since
there is some configuration required to write down our preferences (e.g., "Add versions greater than this that match this regular expression") you'll find uptodate.yaml
files scattered across the repository with these preferences.
The root directory they are found in determines their scope (and the content they will be updating or parsing).
The two keys you will see in the file are dockerhierarchy
and dockerbuild
for the last two bullets.
More detail of how this works in context of our needs to build base and higher level images is detailed below.
A core base image is considered the lowest level - an operating system with only a handful of additional dependencies that won't vary with the operating system. Since the lab heavily uses spack and it makes installing software easy, we also install spack. For comparison with the previous axom repository, Currently, the following directories hold base images:
In the uptodate.yaml
here you'll find the dockerhierarchy
key, which means we always want
to maintain subfolders that correspond with the latest tags of the ubuntu image.
A folder that does not correspond to a tag (e.g., clang) is a matrix build, and only
organized as such to indicate that it uses the ubuntu base. Matrix builds are discussed next.
A matrix image uses a base image, as described above, to create a matrix of different
builds. This means that the uptodate.yaml
in some root directory will have a dockerbuild
section that specifies how to generate the matrices. How are these triggered? When a matrix
build is done, the build args that are containers are added as labels. Then when a nightly check
is done, we compare the current labels from a config manifest
with the latest tags found for the container build args, and re-build if they are different,
or if there are latest build args found but no labels for it (indicating a change in the image).
You'll also notice, thus, that the Dockerfile found under this root use build arguments. This is a different approach than a core base image because unlike base images that can vary, for matrix builds we want to use a common template with different build arguments.
For any Dockerfile in the repository that has a FROM statement (that doesn't use a build arg) we also check for updated hashes for that particular tag. For build arguments that follow any of these patterns we can also easily update versions of build args that are versions from spack, GitHub repository releases or commits, or container identifiers.
After an image is built and deployed, an entry is generated for the library interface. This includes names, versions, and links to metadata to further inspect or interact with the images. The interface also exposes a RESTful endpoint in case you'd like to interact programatically.
A base image is some derivative of an existing OS image (e.g., ubuntu or alpine from Docker Hub) that you want to add some special sauce to, like some editors, compilers, or spack. To add a base image you should ask and answer the following questions.
FROM
that we extend on maps to ubuntu, and a similar ubuntu image with nvidia drivers is nvidia-ubuntu. Dockerfile
s like this for base images because each is different enough to make it hard to template.ubuntu/uptodate.yaml
). You'll want to define a docker hierarchy that tells the nightly updater how to look for new tags (versions) that will be created as new directories. As an example, see below.dockerhierarchy:
container:
name: ubuntu
filter:
# we only want XX.04 versions
- "^[0-9]+[.]04$"
startat: "16.04"
# Versions to skip (not LTS releases)
skips:
- "17.04"
- "19.04"
- "21.04"
A matrix image is what you want when you have a particular base image that you want to iterate some number of versions / variables for, and this means using a common Dockerfile. E.g., "I want to use the base ghcr.io/rse-ops/ubuntu:20.04 and build with three versions of library X. To add a matrix build you should:
dockerbuild
in the uptodate.yaml for an example.Here is an example of an uptodate.yaml that will create a build matrix of ubuntu versions and spack packages:
dockerbuild:
build_args:
cuda_version:
key: cuda
versions:
- "10.1.243"
- "11.0.3"
- "11.1.1"
- "11.2.2"
- "11.3.1"
- "11.4.0"
# Look for ubuntu versions for our base builds
ubuntu_version:
key: ubuntu
name: ghcr.io/rse-ops/ubuntu
type: container
startat: "20.04"
filter:
- "^[0-9]+[.]04$"
For more details on creating a docker build matrix, see the uptodate docs.
And if there is another type of build you want to add not represented here, please open an issue.
Copyright (c) 2017-2021, Lawrence Livermore National Security, LLC. Produced at the Lawrence Livermore National Laboratory.
RADIUSS Docker is licensed under the MIT license LICENSE.
Copyrights and patents in the RADIUSS Docker project are retained by contributors. No copyright assignment is required to contribute to RADIUSS Docker.
This work was produced under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.