search

nebari-dev / nebari-docs

đź“– Documentation for Nebari
https://www.nebari.dev
BSD 3-Clause "New" or "Revised" License
14 stars 29 forks source link

Document how to recover from `Error: another operation (install/upgrade/rollback) is in progress` #475

Open krassowski opened 5 months ago

krassowski commented 5 months ago

If the helm chart upgrade is interrupted the Error: another operation (install/upgrade/rollback) is in progress operation persists preventing further deployment. It should be possible to recover from this by executing a helm rollback command. I do not know how to execute this command

Cf- https://stackoverflow.com/questions/71599858/upgrade-failed-another-operation-install-upgrade-rollback-is-in-progress

krassowski commented 5 months ago

In a local test deployment what worked for me was running:

kubectl get secret -A | grep jupyterhub

This returns a list like

dev              nebari-jupyterhub-sftp                                              Opaque                                2      8d
dev              nebari-jupyterhub-ssh                                               Opaque                                1      8d
dev              sh.helm.release.v1.jupyterhub-dev.v1                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v2                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v3                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v4                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v5                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v6                                helm.sh/release.v1                    1      7d
dev              sh.helm.release.v1.jupyterhub-dev.v7                                helm.sh/release.v1                    1      1d

then find the the one with the greatest version (in this example sh.helm.release.v1.jupyterhub-dev.v7), then running:

kubectl delete secret sh.helm.release.v1.jupyterhub-dev.v7
viniciusdc commented 2 weeks ago

While the above solution only addresses a specific case for JupyterHub (though those files exist for each release), I suggest the same approach you found in that Stack Overflow answer. In K9s, you can inspect Helm releases and installed services by navigating to :helm.

Be cautious when rolling back changes. Depending on the version you're reverting to, you might lose resources specific to that version—for example, a volume that was integrated in a newer release.