bookingcom / shipper

Kubernetes native multi-cluster canary or blue-green rollouts using Helm
Apache License 2.0
734 stars 38 forks source link

Forget items with wrong clusters from the queues #329

Open parhamdoustdar opened 4 years ago

parhamdoustdar commented 4 years ago

The user doesn't specify clusters by name. It's Shipper which decides the clusters that a Release should end up on. So, when we are processing an item and a Cluster is not found, one of two things could be happening:

  1. A user or an automated process has messed with our annotations, or
  2. The Cluster object has been removed.

In either case, this object is not valid and should be removed from the queues instead of being retried.

However, in the second case, there is a chance that the Cluster was removed in error. In that case, the Cluster object will be re-created by a user or an automated process. When that Cluster object is added, we can scan the existing Releases, find Releases that had been scheduled on a cluster with the same name, and enqueue those Releases.