search

svalinn / DAGMC

Direct Accelerated Geometry Monte Carlo Toolkit
https://svalinn.github.io/DAGMC
Other
100 stars 63 forks source link

Reduced number of Dockerfiles #813

Closed shimwell closed 2 years ago

shimwell commented 2 years ago

Description

This PR aims to simplify the CI process from many dockerfiles and scripts to less dockerfiles and scripts, carrying on the good work @bam241 made migrating from Dockerhub and Circle to GitHub Actions.

This is the first part of simplifying the CI, if this PR is successful then the next stage would be to move .sh scripts inside the Docker container

Motivation and Context

Simple is nice

Changes

Multiple Dockerfiles replaced with one

Behavior

Same end point, we should just have an easier system for the CI

shimwell commented 2 years ago

Tests are passing :tada: and I think this is ready for a review

What do people think is this a reasonable first step for simplifying the dockerfiles.

There is more to be done but I am keen to keep the PR easy to review

gonuke commented 2 years ago

Maybe we could temporarily add a master high-level TODO list as a comment in one of these files so that everyone sees the end goal/vision?

shimwell commented 2 years ago

Yep good point, I shall keep an eye on efficiency of the building, looking at the action logs this PR build takes about the same amount of time as previously, ~23mins

For the high level TODO list I was thinking it could be something like this

gonuke commented 2 years ago

Oooo... you could make a GitHub project with these steps defined there... 😀 rather than in a single PR or comment in a file.

gonuke commented 2 years ago

As for efficiency, if it is just as fast, I'd like to make sure we understand why so that we don't accidentally break it in the future.

shimwell commented 2 years ago

As for efficiency, if it is just as fast, I'd like to make sure we understand why so that we don't accidentally break it in the future.

The current builds use -f for file and that file has a few docker files below which also needed to be built. If the images are built and in the local docker image collection then they are used, if they are not found then are downloaded.

This build uses --target which also needs the stages below to have previously been built. If they are in the local collection then they are used and if they are not found then they built from scratch. However the images are found in the cache as the building route starts from the base, tagging each build as it goes and works up the stages.

Each docker image build results in a tagged image and if the same dockerfile is built with the same build args values then the previously tagged image will be used. This happens automatically for modern docker installs on your desktop (buildkit required). Enabling Docker Layer Caching on GitHub requires a bit of extra lines in the existing yml scripts using the setup-qemu-action to get docker layer caching working for the current build scripts and this also applies to the proposed layout.

shimwell commented 2 years ago

Oooo... you could make a GitHub project with these steps defined there... grinning rather than in a single PR or comment in a file.

Yes certainly, I have added the tasks to a project here https://github.com/svalinn/DAGMC/projects/6

gonuke commented 2 years ago

As for efficiency, if it is just as fast, I'd like to make sure we understand why so that we don't accidentally break it in the future.

The current builds use -f for file and that file has a few docker files below which also needed to be built. If the images are built and in the local docker image collection then they are used, if they are not found then are downloaded.

This build uses --target which also needs the stages below to have previously been built. If they are in the local collection then they are used and if they are not found then they built from scratch. However the images are found in the cache as the building route starts from the base, tagging each build as it goes and works up the stages.

Each docker image build results in a tagged image and if the same dockerfile is built with the same build args values then the previously tagged image will be used. This happens automatically for modern docker installs on your desktop (buildkit required). Enabling Docker Layer Caching on GitHub requires a bit of extra lines in the existing yml scripts using the setup-qemu-action to get docker layer caching working for the current build scripts and this also applies to the proposed layout.

If I understand correctly, then, it's the setup-qemu-action part that is key, because we push these images with varying tag names and I'm not convinced that relying on fetching them would work. Instead, we have to rely on the cache.

shimwell commented 2 years ago

If I understand correctly, then, it's the setup-qemu-action part that is key, because we push these images with varying tag names and I'm not convinced that relying on fetching them would work. Instead, we have to rely on the cache.

I agree

pshriwise commented 2 years ago

Just returned from vacation so I'll keep an eye out for the next PR that continues this work. Thanks for diving into it @shimwell!