Feature: Adjust GitHub prune workflow so that it doesn't delete multi-arch release images

chang-annie commented 1 month ago

Description

The Github prune action which clears out our older Docker images is too aggressive and as a result, the release images are being deleted.

This ticket is to adjust the workflow so that the multi-arch release images are no longer being deleted as part of the prune process.

DOD

Acceptance criteria required to realize the requested feature

[x] Uncomment schedule for testing purposes
[x] Update prune workflow so that we're identifying SHAs of any multi-platform release images and exclude them from the clean up process
[x] Re-schedule the prune action and re-enable it
[x] Stretch goal: clean up older tagged images - only want to keep latest snapshot images and all release images
[ ] Stretch goal: avoid pruning patch versions - for example, if current development version is 1.9.0-SNAPSHOT and we're also working on a patch version 1.8.1-SNAPSHOT at the same time, then we would want to keep the images for both snapshot versions

Test Strategy/Script

OTS:

manually kick off the prune workflow (keep dry-run: true)
Use the workflow logs for the prune_ghcr (image-name) steps to confirm the following

The Prune ghcr > Runtime configuration section should include exclude-tags: 1.7.0,1.8.0. For example, using aissemble-vault:

project owner               boozallen
repository                  aissemble
package                     aissemble-vault
exclude-tags                1.7.0,1.8.0
delete-untagged             false
keep-n-tagged               2
dry-run                     true
validate                    false
log-level                   INFO

Review the Prune ghcr > Loaded Package Data section - note the two most recent NON-release versions found. For most images, this should be 1.9.0-SNAPSHOT and 1.8.0-rc8.
NOTE: aissemble-airflow, aissemble-kafka, and aissemble-mlflow images are deprecated and will not have a 1.9.0-SNAPSHOT tag
Review the Prune ghcr > Finding tagged images to delete, keeping 2 versions section and confirm that the list does not include any release versions (e.g., 1.8.0,1.7.0) or the two most recent non-release versions (e.g., 1.9.0-SNAPSHOT, 1.8.0-rc8)
Review the Prune ghcr > Deleting packages section and confirm that the list does not include any release versions or the two most recent non-release versions

Final Test:

Confirm Prune workflow is re-enabled
Kick off the prune workflow with the dev branch and confirm successful
Use the workflow logs for the prune_ghcr (image-name) steps to confirm the following

The Prune ghcr > Runtime configuration section should include exclude-tags: 1.7.0,1.8.0. For example, using aissemble-vault:

project owner               boozallen
repository                  aissemble
package                     aissemble-vault
exclude-tags                1.7.0,1.8.0
delete-untagged             false
keep-n-tagged               2
validate                    false
log-level                   INFO

Review the Prune ghcr > Loaded Package Data section - note the two most recent NON-release versions found. For most images, this should be 1.9.0-SNAPSHOT and 1.8.0-rc8.
NOTE: aissemble-airflow, aissemble-kafka, and aissemble-mlflow images are deprecated and will not have a 1.9.0-SNAPSHOT tag
Review the Prune ghcr > Finding tagged images to delete, keeping 2 versions section and confirm that the list does not include any release versions (e.g., 1.8.0,1.7.0) or the two most recent non-release versions (e.g., 1.9.0-SNAPSHOT, 1.8.0-rc8)
Review the Prune ghcr > Deleting packages section and confirm that the list does not include any release versions or the two most recent non-release versions

Successfully pull a 1.8.0, 1.7.0, and 1.9.0-SNAPSHOT docker image

docker pull ghcr.io/boozallen/[image-name]:1.8.0
docker pull ghcr.io/boozallen/[image-name]:1.7.0
docker pull ghcr.io/boozallen/[image-name]:1.9.0-SNAPSHOT

References/Additional Context

Container retention policy doc

chang-annie commented 1 month ago

DOD completed with @ewilkins-csi

chang-annie commented 3 weeks ago

Some interesting findings:

When trying to pull on the manifest SHA for the 1.8.0 release images, we cannot simply use something like:

docker buildx imagetools inspect ghcr.io/boozallen/aissemble-spark:1.8.0 --format "{{json .Manifest}}" | jq -r .digest

This will output:

ERROR: failed to copy: httpReadSeeker: failed open: content at https://ghcr.io/v2/boozallen/aissemble-spark/manifests/sha256:c0ea773c38265bf3a80a133211aba2468dec87b8ce5b341d82611e2c81252147 not found: not found

Where the aforementioned SHA is connected to the build attestations. These are created because we have buildx's provenance set to min aka not false (default setting). This creates two extra manifests in the manifest list with os: unknown and architecture: unknown. You can further inspect those manifests to retrieve basic provenance info.

chang-annie commented 3 weeks ago

OTS completed with @carter-cundiff and @jaebchoi

ewilkins-csi commented 2 weeks ago

OTS passed ✅

J-Clingerman commented 2 weeks ago

All tests passed. ✅

chang-annie commented 2 weeks ago

Reopening so we can test the fix to address docker images with no release versions (which is adding an extra comma at the front of our exclude-tags parameter - e.g., exclude-tags: ,1.7.0,1.8.0)

cwoods-cpointe commented 2 weeks ago

Final test: Passed. Did not delete any release version or the 2 latest dev versions.

boozallen / aissemble