search

StackStorm / stackstorm-k8s

K8s Helm Chart that codifies StackStorm (aka "IFTTT for Ops" https://stackstorm.com/) Highly Availability fleet as a simple to use reproducible infrastructure-as-code app
https://helm.stackstorm.com/
Apache License 2.0
105 stars 107 forks source link

Additional Steps for Upgrading HA Chart to the Latest Version #278

Closed wantdrink closed 2 years ago

wantdrink commented 2 years ago

Hi there,

We plan to upgrade our HA chart from 0.32(stackstorm 3.3) to 0.8(stackstorm 3.6). We want to keep using the external Mongo/Redis/RabbitMQ we've installed for SS.

I'd like to know is there any additional step needed for this upgrade? Looks like the only thing we need to do is modify the values based on the latest chart's requirement and run helm upgrade with the specified version is enough.

Thanks a lot.

arm4b commented 2 years ago

We don't have version-to-version migration scripts. However, you can look at the Changelog and https://github.com/StackStorm/stackstorm-ha/releases for each version, while comparing the Helm chart values and its structure. Please also pay more attention to the breaking changes in the Changelog.

wantdrink commented 2 years ago

Thanks @armab. I prepared to compare the difference of values.yaml and update to the latest format/content before upgrading. So if the changed values.yaml is correct and fit for the latest chart, after executing helm upgrade, the chart will download stackstorm 3.6 images and create/update databases as needed, we don't need to do additional change manually right? thanks.

arm4b commented 2 years ago

@wantdrink With so many changes between those versions it's best to double-check. Perhaps creating a testing st2 deployment first with your existing Helm values configuration, installing there 0.32, and then performing an upgrade to 0.8 will give more ideas about what to expect in the production.

wantdrink commented 2 years ago

Thank you @armab. I'll try it and update here.

cognifloyd commented 2 years ago

@wantdrink How did it go?

I'm going to close this in March unless you need clarification on something.

wantdrink commented 2 years ago

@cognifloyd Sorry for my late. I just upgraded it last week. I changed the old version's values.yaml to fit for the latest one. After that if running upgrade directly it will tell that the deployment was failed. Looks like there is a little difference between the old one and the latest version for the resource type deployed as pods.

I simply delete the previous helm release and then install a new one with new yaml. Everything looks good. No DB script needs to be executed so the DB update might be included in code.

Finally for my case delete the helm release and install with the new chart & value yaml should be ok. The flows/actions/history are still in mongo DB. The only issue is that in PROD env maybe we need to stop serving for several minutes.

Thanks again.

cognifloyd commented 2 years ago

Yup. I went through a similar migration when I migrated from the pre-helm-chart installation to using the helm chart. I tested my values in a different cluster to ensure I had things working correctly. Then, I announced that there'd be a maintenance window for StackStorm and updated in that time.

I don't remember how long that took me - but I had a lot of moving pieces to upgrade including the st2 version, the mongo version (external to k8s cluster), the rabbitmq version (external to k8s cluster), the load balancer (nginx) config (external to k8s cluster), custom secrets naming, and a few other things.

Just review (and if at all possible test!) your values carefully to ensure you've updated them. I recommend using git to compare the values.yaml file between chart version you were using with the latest chart. git provides a much more detailed view of changes than the changelog should provide.

Bumping chart versions should be much simpler once we pass v1.0.

Good Luck on the upgrade! I'm going to close this now, but feel free to submit PRs or file issues for any improvements that make/made your upgrade easier. Our slack channels and discussions are also great places to get more help if you need it.

wantdrink commented 2 years ago

Thanks a lot @cognifloyd . So far it runs more than 2 weeks after upgrading. Everything looks good to me.