Graceful Pack Upgrades

[LindsayHill]

Several $$ customers have been asking about how to gracefully handle rolling out upgrades to workflows and actions. We have some of the pieces in place with content_version, but this doesn't handle things like rolling out a new version of an Orquesta workflow.

Customer concerns include:

• What happens to running workflows when I roll out an upgrade? At what point does the new workflow get read? Workflow only read when workflow starts? What if it's updated half-way through?
• How can I gracefully drain workflows prior to rolling out an upgrade? How do I ensure that all current instances of old workflow complete prior to running new one?

We need to: (A) Provide guidance to customers about current behavior when actions and workflows are updated (B) Provide guidance about how best to safely roll out updates (C) Review the above, and see if we need to make improvements - e.g. put the system in some sort of 'graceful drain' mode, where currently executing workflows are finished, but no new workflows can begin. Once existing workflows have finished, load new workflows, then start accepting jobs again.

We should document (A) & (B) first, so we can plan our next moves.

[arm4b]

I'd suggest to document Blue/Green deployments at a high level, - it's something that @bigmstone did for our st2cicd.

• st2cicd is placed behind a load balancer.
• Deploy new StackStorm instance.
• Do some automated checks if new st2 works OK.
• Switch Load Balancer to point to new st2.
• Disable all rules/triggers on old st2 and wait for workflows to finish.
• Delete old st2 instance.

It provides more production-level benefits like possibility to revert to old deployment if something went wrong with new st2. However it brings more burden to Ops team who manages this deployment.

Of course it's best to implement something in st2 core for pack upgrades which is a good thing, but here is ^^ at least something.

[warrenvw]

Reference: https://github.com/StackStorm/st2enterprise-dockerfiles/issues/3