search

operator-framework / operator-controller

Apache License 2.0
29 stars 47 forks source link

operator-controller should not uninstall/rollback a failed install/upgrade #994

Open joelanford opened 1 week ago

joelanford commented 1 week ago

Right now, operator-controller is using helm-operator-plugins, which automatically uninstalls failed installations and rolls back failed upgrades.

This is problematic because the installation or upgrade may have progressed to the point of no return. For example:

In my opinion, a better behavior is to just stop, inform a user of the problem, and require human intervention to resolve and progress.

joelanford commented 5 days ago

I have a change in helm-operator-plugins that adds a knob to turn this behavior off. From a PoC standpoint, this looks very straightforward.

joelanford commented 4 days ago

PR up to make it possible for operator-controller to configure this behavior in helm-operator-plugins: https://github.com/operator-framework/helm-operator-plugins/pull/348

bentito commented 3 days ago

I think some high-frequency environments, like particularly those managed by SD, are going to want the simplicity of the default Helm rollback behavior. As long as we leave the knob for them to allow automatic rollback on failed upgrade, that should be enough.

joelanford commented 4 hours ago

I'm not sure I agree. I think the important thing for a managed service SRE user is that:

  1. The failure is apparent
  2. There is a way to centrally apply a fix and have that fix propagate to the affected installations/upgrades.

If OLM automates rollbacks, it could make an SRE's job harder because it could make the problem worse (as described in the description), and it could make troubleshooting more difficult because logs/metrics/object status in the failed release would be wiped out by the rollback.