openculinary / infrastructure

This repository documents the steps required to set up a fresh RecipeRadar environment
GNU Affero General Public License v3.0
5 stars 5 forks source link

Documentation: add a process for cleaning up stale container images #46

Open jayaddison opened 1 month ago

jayaddison commented 1 month ago

Is your feature request related to a problem? Please describe. During a recent ad-hoc cleanup of stale container images, some current (in-production) container images were mistakenly deleted, and this eventually caused the website outage described in #44.

We should determine and document a process to safely clean up stale container images.

Describe the solution you'd like Documentation, including command-line examples, about how to remove stale container images as an operational process in the README.md file of this repository.

Describe alternatives you've considered N/A

Additional context Arises from prevention measures discussed in #45.

jayaddison commented 2 weeks ago

I'd suggest that we retain at least two images for each Kubernetes deployment:

One complexity for the latter is that we recently began omitting timestamps from the container images themselves (e.g. https://github.com/openculinary/api/commit/db144f40a6a14812b20997032a56985fadaf5b4c). We could however determine the next-most-recent image by referring to the source control history of the main branch (starting from the currently-deployed commit ID).