Provide a script to cleanup old containers from the Azure Container Registry

pim-simons commented 2 years ago

Is your feature request related to a problem? Please describe. Currently there is not an easy way to cleanup old containers from the Azure Container Registry.

Describe the solution you'd like A script to cleanup/untag images from the Azure Container Registry if the image is no longer found in the revision history of AKS across environments.

pim-simons commented 2 years ago

This will be based on the scripts of @bdestercke, he will also assist in development so really all credits go to him 🥇

pim-simons commented 2 years ago

@stijnmoreels what would be a good place for this/these script(s)? Maybe Arcus.Scripting.Management as it does have something to do with managing resources. Or maybe an entirely new module, something like Arcus.Scripting.ACR? Interested in your view on this 👍🏻

stijnmoreels commented 2 years ago

Ok, great stuff! Super that we can again have a great idea from our contributors. A good question, where should we put it. Hmm, We created the 'Management' module because the 'soft deleted' ones are not necessary related to the type of resource they're deleting, right? Or am I wrong? Otherwise they would be in our API Management module. So, I'm wondering if this indeed needs it's own module, or if it matches a similar purpose as the 'Management' module. Maybe, (and that's just my limit understanding of ACR), it would indeed be better to have this in a separate module, as it's more 'reasonable' if one has to look for it. It still has to do with Azure Container Registry specific, right? The images, the history... But again, I'm no expert. Maybe if the script comes to completion, and we have a PR, we can see if a separate module is the right place or not.

pim-simons commented 2 years ago

Ok, great stuff! Super that we can again have a great idea from our contributors. A good question, where should we put it. Hmm, We created the 'Management' module because the 'soft deleted' ones are not necessary related to the type of resource they're deleting, right? Or am I wrong? Otherwise they would be in our API Management module. So, I'm wondering if this indeed needs it's own module, or if it matches a similar purpose as the 'Management' module. Maybe, (and that's just my limit understanding of ACR), it would indeed be better to have this in a separate module, as it's more 'reasonable' if one has to look for it. It still has to do with Azure Container Registry specific, right? The images, the history... But again, I'm no expert. Maybe if the script comes to completion, and we have a PR, we can see if a separate module is the right place or not.

You make some valid points, I will stick with Arcus.Scripting.ACR as that seems like the most logical one for now, thanks for your insights 👍🏻

stijnmoreels commented 2 years ago

Anytime 😉 , excited that we have a new contributor! 🥳

pim-simons commented 1 year ago

@bdestercke how does https://learn.microsoft.com/en-us/azure/aks/image-cleaner?tabs=azure-cli relate to the functionality you require? This is still in preview, but if it supplies the required functionality than we might not need to create our own script for this.

bdestercke commented 1 year ago

@pim-simons unfortunately that functionality only does a cleanup of the images on the nodes and doesn't touch ACR

fgheysels commented 1 year ago

Is there a possibility too detach the AKS funtionality from this ? As I can see there might be other usecases for this when you do not use AKS.

bdestercke commented 1 year ago

For this case we're trying to have the AKS as the reference source of the images that we need to keep. (so we don't remove images that are actively used or referenced) Maybe it might be nice to set it up so that we can have multiple sources that add to the list of images that need to be kept and afterwards this file can be used to run a cleanup on acr by removing all images that are not in this file/list.

fgheysels commented 1 year ago

Just a wild idea, but maybe we just need to feed the names of the images that must be removed. We can have another script / tool that lists the images that can be removed by inspecting AKS as a reference source. So in fact, we split up this one script into 2 separate scripts, where each script has it's own task. This is a little bit the unix way of thinking.

This should also make it possible to easily create new scripts that inspect other reference sources.

bdestercke commented 1 year ago

I currently have 4 scripts:

Get-ContainerVersionTags.ps1: gets all the tags and repositories from a kubernetes cluster / namespace and writes these to a file. (images we need to keep since they are still in use)
Get-MultiContext-ContainerVersionTags.ps1: helper script to run Get-ContainerVersionTags.ps1 against multiple kubernetes contexts/namespaces.
Untag-Images.ps1: script to untag the images that are not listed in the file created by Get-ContainerVersionTags.ps1 and are older than x amount of days.
Purge_Untagged_Manifests.ps1: removes all the manifests that have no tags associated with them from the ACR.

pim-simons commented 2 months ago

After offline discussion we are closing this task, we have other ways of cleaning the ACR (acr purge) that covers most of the requirements here. We can always re-open if we find this functionality is something we need later on.

stijnmoreels commented 2 months ago

Thx for the investigation! 👍

arcus-azure / arcus.scripting

Provide a script to cleanup old containers from the Azure Container Registry #299