uselagoon / remote-controller

A group of controllers for handling Lagoon builds and tasks in Kubernetes or Openshift
5 stars 1 forks source link

fix: cache cancellations to prevent race conditions in pod creation #230

Closed shreddedbacon closed 1 year ago

shreddedbacon commented 1 year ago

Checklist

There exists a condition where a cancellation can be actioned before the task or build has been started, so that when the task or build are started they have no idea that it was cancelled. This is because there are two queues used, one for builds and tasks, the other for misc tasks. Cancellations enter via the misc queue, and because they are handled in their own queue, it is possible that the cancellation message could be processed before the build or task message is handled completely.

This implements an expiring cache that will cache cancellations, so that if the condition is met and the pod gets started, the pod controller will check the cache periodically and if the cancellation exists in cache will cancel the task or build in the way it would normally be cancelled.