search

operator-framework / operator-lifecycle-manager

A management framework for extending Kubernetes with Operators
https://olm.operatorframework.io
Apache License 2.0
1.72k stars 545 forks source link

What is the best practice to no-down-time upgrade OLM itself? #2318

Open zhaminglei opened 3 years ago

zhaminglei commented 3 years ago

A.T.,

We are looking for a way to keep OLM itself updated, for example, we have 0.18.1 running and now we want to update OLM to 0.18.3, how can we do this without turn down the services running on top of OLM?

let me know if any other context needed, thanks a lot!

dmesser commented 3 years ago

Can you elaborate how the services depend on OLM and what makes the update of OLM itself noticeable? An OLM update is a fairly straight replacement of controllers with new images, the "down time" is really just time in which objects will not be reconciled and that should be rather short.

zhaminglei commented 3 years ago

Can you elaborate how the services depend on OLM and what makes the update of OLM itself noticeable? An OLM update is a fairly straight replacement of controllers with new images, the "down time" is really just time in which objects will not be reconciled and that should be rather short.

Thanks for replying, one of examples is that we implemented a software self upgrade web UI based on InstallPlan, aka, a service is detecting the crd installPlan, if there is an approved installPlan and a installed CSV with same version, we show an entry to the UI that this version was installed. If there is an unapproved installplan, we show an entry to the UI that there is a newer version available, and then the UI can approve the install plan to install new version.

As you can see, the feature is based on CRD "installplans.operators.coreos.com" and "clusterserviceversions.operators.coreos.com" that are part of OLM, if we uninstall the OLM and re-install OLM, I think the resources will be deleted since the CRD was deleted and re-installed, let me know if I am missing anything.

kevinrizza commented 3 years ago

if we uninstall the OLM and re-install OLM

How are you actually uninstalling OLM? Ideally, when you upgrade OLM to a newer version, you should be able to just apply the new set of resources. OLM supports leader election, so applying changes to the set of deployments and CRDs between versions should be enough to upgrade OLM without actually removing anything. At the very least, you shouldn't need to delete the CRDs (and the related data loss) in order to upgrade OLM.

BostjanBozic commented 3 years ago

In the past, I just rerun the install.sh script and that performed the update (since it is using kubectl apply. But with recent version, following was added to script:

if kubectl get deployment olm-operator -n ${namespace} > /dev/null 2>&1; then
    echo "OLM is already installed in ${namespace} namespace. Exiting..."
    exit 1
fi

I guess this would make sense if someone customised OLM deployment, but otherwise this breaks previous "upgrade" path.