search

aws-samples / aws-deployment-pipeline-reference-architecture

Reference architecture for deployment pipelines
Other
284 stars 47 forks source link

Hotfix/Emergency Deployments #16

Open scubbo opened 1 year ago

scubbo commented 1 year ago

Love this reference and the clear terminology, it's going to make focused conversations about approaches a lot easier - thank you!

I'm curious how a hotfix/emergency-deploy system would fit into this system? Obviously, we hope to never need them, but hope is not a reliable mechanism. Do you build add-on tooling which can make an ad-hoc deployment to a target stage of the output of a given build-iteration, or is there a way to build that functionality into the pipeline itself? Having left Amazon recently, I'm wistfully remembering the extra functionality the internal Pipelines tool gives...

cplee commented 1 year ago

@scubbo - are you looking to skip the earlier environments and have a commit target a higher environment directly?

scubbo commented 1 year ago

Yep! Obviously, in an ideal world, all changes would flow through the safety-check mechanisms earlier in the pipeline (and I've seen my fair share of situations where a "hotfix" actually made things worse) - but for a lengthy pipeline and imagining a high-impact issue that is not resolvable via a rollback, a "breakglass" mechanism to say "as soon as this is built, deploy it to Prod" would be a useful thing to have, even if we hope to never use it.

scubbo commented 1 year ago

Any progress on this?

cplee commented 1 year ago

No progress to report at this time