Documentation: add an incident report for the 2024-08-10 site outage

[jayaddison]

Is your feature request related to a problem? Please describe. The RecipeRadar service was unavailable for a significant duration of time from 2024-08-10 until recovery was completed on 2024-08-12. We should add a writeup of the timeline, identified problem cause(s), and suggested adjustments to prevent this happening again.

Describe the solution you'd like An incident report in the docs/incidents directory of this repository.

Describe alternatives you've considered N/A

Additional context N/A

[jayaddison]

Approximate timeline; details to be confirmed where possible:

• 2023-06-19: CRI-O tooling updated to v1.27
• Late July / early August 2024: production hosting upgraded to use Ubuntu 24.04
• Early August 2024
  • AppArmor problems noticed during teardown of containers -- permission denied when terminating container monitoring processes
  • Manual cleanup of processes is used to work around the problem.
  • Further deployments take place.
  • System journal error messages begin to accumulate -- a mixture of failures to teardown containers, and failure to reclaim resources from manually cleaned-up containers.
• 2024-08-10: Disk space in production is exhausted due to the container teardown/resource reclamation error messages.
• 2024-08-12
  • The site outage is noticed by operational staff.
  • Disk space is identified as the problem; space is reclaimed and service restoration begins.
  • Operational staff decide to opportunistically upgrade the container runtime, given the identification of the error messages that seem a probable cause of the outage.
  • The system is recovered, and subsequent deployments are able to teardown container resources sucessfully.

Various things that went wrong:

• Lack of system uptime monitoring caused a delay identifying the outage.
• Errors that had been observed and noticed by staff during deployment of containers were ignored.
• A non-scalable manual cleanup approach was implemented instead of identifying and resolving a production problem.
• An outdated container runtime stack was in use in production.

What went well:

• During recovery, the deployment-time container teardown problem was solved (albeit at some risk; outage recovery isn't a great time to upgrade dependencies).

In terms of root cause: it seems that the fact that the deployed CRI-O container runtime in production was out-of-sync with the AppArmor rules from Ubuntu 24.04 is what ultimately resulted in the problem occurring.

We should encourage and make time for clearing up errors that appear in production -- generally we want the system logs and journals to be fairly clean and minimal, so that problems are easier to identify when they occur. This shouldn't involve egregious filtering of logs; instead it should generally involve fixing and reconfiguring our code and components so that fewer unnecessary info/warning/error level messages are produced.

[jayaddison]

Late July / early August 2024: production hosting upgraded to use Ubuntu 24.04

In terms of root cause: it seems that the fact that the deployed CRI-O container runtime in production was out-of-sync with the AppArmor rules from Ubuntu 24.04 is what ultimately resulted in the problem occurring.

This seems incorrect: the Ubuntu 24.04 upgrade occurred on 2024-08-12 as part of the opportunistic updates during outage recovery.

So it may be the case that an earlier update to the AppArmor rules caused the breakage. This may still be a dependency-version-skew problem (rules and runtime out-of-sync), but the way it was introduced seems less clear.

[jayaddison]

Resolved by #43.