Replace ECR push in GHA with ECR pull-through cache

aethr commented 1 month ago

The problem

Current approach to getting a setup-wrf container in AWS ECR is to pull the built image from GHCR and then push it up to ECR in CI. However, this requires adding AWS credentials to GitHub and adds time and complexity to the CI pipeline.

AWS ECR supports pull through caching of images: https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html. When fetching the image through ECR, if the upstream registry is GHCR, ECR will fetch and cache the image.

Definition of "done"

[ ] Docker images from setup-wrf are available in ECR
[ ] ECR push build step has been removed from GitHub Actions
[ ] AWS secrets and account details have been removed from the repo

Additional context

aethr commented 1 month ago

Relevant part of the CI config that we want to remove: https://github.com/openmethane/setup-wrf/blob/main/.github/workflows/build.yaml#L184-L249

lewisjared commented 1 month ago

We used to use the pull through cache but it didn't work for our use case. The pull through cache only updates at most once per 24 hours. While this works fine with immutable tags, it doesn't work with mutable tags that may change at a cadence higher than that (e.g. pr tags or latest which might change multiple times a day). It lead to confusing behaviour as you were never sure what image you would get.

aethr commented 1 month ago

@lewisjared that's good context.

I think once the pace of active development has slowed down, pinning containers to non-ephemeral versions is a reasonable practice. Even if we didn't, I'm not sure we'll have the urgency to need a "latest" tag to be the most recent when the next job runs.

All in all, removing complexity from the repo may still be a worthwhile tradeoff. Will leave this open for now, happy to discuss.

openmethane / setup-wrf