Mention rolling update strategy is not appropriate for Crate major version upgrade

orchestracities / charts

This project includes helm charts developed to deploy orchestra cities (thus covering as well some FIWARE services)

Apache License 2.0

11 stars 9 forks source link

Mention rolling update strategy is not appropriate for Crate major version upgrade #81

Open c0c0n3 opened 4 years ago

c0c0n3 commented 4 years ago

Our Crate chart instantiates a K8s statefulset with a rolling update strategy. With this strategy in place, on detecting a new Crate image, K8s will delete and then recreate each pod with the new image, one at a time. So while upgrading, most of the time you'll be in a situation where some of the pods are running the old image whereas some others are on the new. Crate docs say this is fine only when upgrading to a new patch version whereas to upgrade to a new minor or major version you should first shut down the entire cluster, then upgrade images, and finally bring the new cluster live. This latter procedure isn't compatible with K8s rolling update strategy so we should warn users about it. For example we could mention it in the chart README.

amotl commented 3 years ago

Dear @c0c0n3,

by following up on our conversation at https://github.com/smartsdk/ngsi-timeseries-api/pull/384#issuecomment-731328901 ff. - FWIW, I would also like to reference https://github.com/crate/crate-operator/issues/127 here, where I asked back about whether topics mentioned here might be implemented through the CrateDB Kubernetes Operator or not.

With kind regards, Andreas.

c0c0n3 commented 3 years ago

Dear @amotl, thank you sooo much for being so proactive. It really helps us alot. I added my comments to crate/crate-operator#127. Many thanks!!

c0c0n3 commented 3 years ago

@chicco785 the bottom line for now is:

K8s rolling upgrade isn't the proper way to upgrade to a new major version number, but should work fine between minor versions
There's a very nice tool the Crate guys have been working on for Crate/K8s deployments: https://github.com/crate/crate-operator
There are plans for the Crate Operator to support Crate major version upgrades: https://github.com/crate/crate-operator/issues/128

We should keep an eye on the above issue, since Crate Operator looks like a very nice option for our prod deployments.

chicco785 commented 3 years ago

@chicco785 the bottom line for now is:

K8s rolling upgrade isn't the proper way to upgrade to a new major version number, but should work fine between minor versions

yep, I think this was clear given current experience

There's a very nice tool the Crate guys have been working on for Crate/K8s deployments: https://github.com/crate/crate-operator

that's very interesting, the issue for us is that we would need to be able to reuse existing pvc, or it will be a pain to migrate data. i suppose it works quite well also with fluxctl

There are plans for the Crate Operator to support Crate major version upgrades: crate/crate-operator#128

We should keep an eye on the above issue, since Crate Operator looks like a very nice option for our prod deployments.

c0c0n3 commented 3 years ago

would need to be able to reuse existing pvc

I think the Crate Operator will leave it be, if I understand the way it works but I could be wrong :-)