When upgrading to a new "bad" version over a "good" version - service is down! Support rollback action

jfrog / kubexray

JFrog KubeXray scanner on Kubernetes

Apache License 2.0

25 stars 9 forks source link

When upgrading to a new "bad" version over a "good" version - service is down! Support rollback action #17

Closed eldada closed 5 years ago

eldada commented 5 years ago

BUG

What happened:

I have a good version running: 1.1
I upgrade it using helm upgrade ..... to bad version 1.2
KubeXray detects the bad version and scales deployment down to 0
Service is down!

What should happen:

KubeXray should block deployment of bad version 1.2 but allow version 1.1 to keep running (rollback)
Service is still up (with old, good version 1.1)

DarthFennec commented 5 years ago

Not really sure what to do about this. The main issue is I don't know of a way to block deployment, or otherwise intercept deployment before it completes. The closest thing I know of is what KubeXray does right now: receive a trigger after a deployment completes, and then modify the deployment to no longer run. This happens after helm is done and thinks everything went well, so there's no way to let it know that something broke and that it should keep the old version up, and KubeXray can't do that on its own because it doesn't have the context to recognize things like helm upgrade.

eldada commented 5 years ago

I understand. A solution should be found to allow service continuity. I will not want to have kubexray running in production and shutting down my service if an issue is found. I want it to block the new version and keep the existing. Maybe save the currently running version (if exists) before any changes are made, so it's able to compare new state to old one? This will require a state to be recorded and "remembered" by kubexray for every running container.

ankushchadha commented 5 years ago

@DarthFennec - one way to intercept requests before the actual deployment is to use admission controllers. Let's discuss more and provide a revised spec for the next version of kubexray.

eldada commented 5 years ago

IMHO - do be used in production, we must implement a rollback option or the adoption of this will be very limited.

ant-caichu commented 5 years ago

In my opinion, we can not rely on helm lifecycle event hook, as not all the people use helm. Kubernetes container hook only support postStart and preStop, so this also can not be a good solution. Admission controller may do something like podSecurityPolicy provide admit mechanism to allow or reject pod run, user can choose if they need it.

rimusz commented 5 years ago

As KubeXray functionally will be ported to Kubenab, I'm closing this issue.