org-formation / org-formation-cli

Better than landingzones!
MIT License
1.4k stars 129 forks source link

Bug: Perform-tasks fails when creating/updating stacks with large templates to 2+ regions #569

Closed kuggis closed 1 month ago

kuggis commented 1 month ago

I'd like to report an edge-case bug that I've come across recently. The following conditions need to be met:

  1. CloudFormation stack template must be sufficiently large that Org-Formation chooses to upload the template to S3 first.
  2. You must be deploying the large template to 2+ regions.

When you run perform-tasks Org-Formation is able to successfully update one of the stacks but fails to update the second stack with an error like:

ERROR: Stack test-log-group-us-east-2 in account 123456789012 (us-east-2) update failed. reason: unable to create bucket org-formation-123456789012-us-east-2-large-templates in account 123456789012, error: IllegalLocationConstraintException: The us-east-2 location constraint is incompatible for the region specific endpoint this request was sent to. (123456789012 = PlaygroundAccount)

Running perform-tasks again is a workaround and will result in the stack in the second region being updated successfully, this issue is not a blocker but it is a nuisance. I ran Wireshark while reproducing the issue and confirmed that indeed Org-Formation is trying to contact a bucket in us-east-2 using an eu-west-1 endpoint.