Change HOME_REGION in global config but AWSAccelerator pipeline still get stuck with us-east-1 at Account stage

[quangchustudy]

Describe the bug

To Reproduce Use the Region selector in the console navigation bar to choose "ap-southeast-1" to deploy the cloud formation template file from the beginning. Take a look at codepipeline at the Account stage

Expected behavior Expect successfully with using "ap-southeast-1" not us-east-1

Please complete the following information about the solution:

• [ ] Version: 1.4.0

SO0199-pipeline

• [ ] Region: ap-southeast-1

Screenshots

Additional context Add any other context about the problem here.

[gavinying]

I had exactly the same issue, thanks for raising it.

[awsclemj]

Hello, and thank you for your interest in Landing Zone Accelerator on AWS!

There are some API calls for global services that must be completed in us-east-1, specifically AWS Organizations API calls in this solution. The Accounts stage is always deployed to us-east-1 for this reason.

us-east-1 is bootstrapped when the solution is installed. If you have any SCPs or region deny that block actions in us-east-1, it’s possible the region may not have been bootstrapped properly. You should see a CDKToolkit stack in that region once the pipeline has completed. If you do not, can you verify you do not have any region deny settings in place for us-east-1 and run the Installer CodePipeline again? The Install stage in that pipeline should bootstrap us-east-1 for you.

More info on regional endpoints for AWS Organizations: https://docs.aws.amazon.com/general/latest/gr/ao.html

Thank you! Please let us know of this resolves your issue.

[quangchustudy]

Hi @awsclemj , @gavinying

Issue already resolved and you're right for "The Accounts stage is always deployed to us-east-1". I got stuck with this problem since I did a clean-up (including s3 bucket cdk-accel-assets-xxxxxxx-us-east-1 but still keep the cloudformation of CDKToolkit stack on us-east-1) for the last deployed in wrong region us-east-1 while I setup landing zone with control tower as https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/prerequisites.html => error happened at Account Stage.

And I solved as following:

• Deleted the old CDKToolkit stack on us-east-1
• Trigger release change to run AWSAccelerator-Pipeline again then you can see the CDKToolkit will be created on us-east-1 Pls noted: above CDKToolkit of Account on us-east-1 is different with CDToolkit on desired region

Thank you.

[gavinying]

@awsclemj @quangchustudy Unfortunately, I still have this issue.

• the problem is AWSAccelerator-Pipeline seems not creating AWSAccelerator-CDKToolkit stack in us-east-1 region (ap-southeast-1 region successfully created stacks).
• I have already added us-east-1 to allowed region;
• I have cleaned up S3 and cloudwatch in both regions before I run AWSAccelerator-Pipeline

My question is,

• is AWSAccelerator-CDKToolkit stack in us-east-1 region supposed to be created during Prepare stage of AWSAccelerator-Pipeline? Why I couldn't see any logs about creating stacks in us-east-1 region while the stage succeded?

[awsclemj]

Hello @gavinying,

us-east-1 is bootstrapped during the Install stage of AWSAccelerator-Installer pipeline. I’d suggested manually releasing that pipeline once again if you have recently made edits to your region deny settings and do not see the stack created.

Thanks!

[gavinying]

Hi @awsclemj , I have re-run the AWSAccelerator-Installer pipeline just now, it completed without error, but I still didn't see any messages about creating stacks in us-east-1 region in the logs. I have attached logs for your reference. log.txt Any other suggestion? thanks.

[awsclemj]

Based on this log message, it appears your global region has been modified:

Container] 2023/05/09 00:42:17 Running command if ! aws cloudformation describe-stacks --stack-name AWSAccelerator-CDKToolkit --region ap-southeast-1; then BOOTSTRAPPED_GLOBAL="no"; fi

The global region is defined here in our installer template, so it must have been modified externally. We strongly advise against modifying the source code of LZA, as it may lead to unexpected failures of the core engine. I would suggest updating your Installer stack using an unmodified template so us-east-1 is bootstrapped.

Thanks!

[gavinying]

@awsclemj Awesome, you saved my day, thank you so much!

[SennaSemakula]

@awsclemj Do you know why this is not part of https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/prerequisites.html. I hit this issue whilst using Control Tower setup by using us-east-2 and denying us-east-1 for regions. The pre-requisites should clearly state that us-east-1 is required