Stack fails with error - Insufficient permissions to invoke AWSCloudFormationStackSetAccountGate

[jbernales5]

Describe the bug The stack fails during the CloudFormation deployment when creating the resource StackSetManagerCustomResource with the following error message Failed to create resource. See the details in CloudWatch Log Stream: 2021/01/05/[$LATEST]861da7d97d0942b28e916e93fc4258d2. The Rollback process is then initiated and the stack resources are deleted.

To Reproduce The README instructions have been followed on two different accounts for testing purposes:

• A non-root account which is part of an organization;
• An individual root account (not part of any organization).

Please note that the issue has been reproduced on both accounts with the same error messages and logs.

The following commands were executed on both accounts:

cd ./deployment
chmod +x ./run-unit-tests.sh
./run-unit-tests.sh

export REGION=eu-west-1
export DIST_OUTPUT_BUCKET=cognito-backup-bucket-test
export SOLUTION_NAME=my-cognito-backups
export VERSION=1

aws s3 mb s3://$DIST_OUTPUT_BUCKET-$REGION --region $REGION

chmod +x ./build-s3-dist.sh
./build-s3-dist.sh $DIST_OUTPUT_BUCKET $SOLUTION_NAME $VERSION

aws s3 cp ./regional-s3-assets/ s3://$DIST_OUTPUT_BUCKET-$REGION/$SOLUTION_NAME/$VERSION/ --recursive --acl bucket-owner-full-control

I then navigated to the CloudFormation console panel, added the cognito-user-profiles-export-reference-architecture.template S3 URL and provided the necessary information (cognito user pool, replication region, etc.)

Expected behavior The CloudFormation template is successfully deployed with no issues.

Please complete the following information about the solution:

• [x] Version: v1.0.0
• [x] Region: Main regions: eu-west-1, eu-central-1. Replication regions: eu-west-2 and eu-west-1 respectively.
• [ ] Was the solution modified from the version published on this repository?: NO
• [ ] If the answer to the previous question was yes, are the changes available on GitHub? N/A
• [x] Have you checked your service quotas for the sevices this solution uses?
• [x] Were there any errors in the CloudWatch Logs?

Logs

• StackSetManagerCustomResource message: Failed to create resource. See the details in CloudWatch Log Stream: 2021/01/05/[$LATEST]7e0d341742314c669d511d4b54cd6626
• Log from StackSetCheckStatusLambda log group 7e0d341742314c669d511d4b54cd6626:

  2021-01-05T11:58:24.754Z    3150424a-8435-4786-a72c-807d3d4e09c8    INFO    List Operation Results Response: 
  {
  "ResponseMetadata": {
      "RequestId": "7f599f9d-6aad-4516-bc32-a7be92213b7d"
  },
  "Summaries": [
      {
          "Account": "464691222504",
          "Region": "eu-west-1",
          "Status": "FAILED",
          "StatusReason": "ResourceLogicalId:StackSetConstantsCustomResourceLambda, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Error occurred while GetObject. S3 Error Code: NoSuchBucket. S3 Error Message: The specified bucket does not exist (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: a488742f-5ff5-4eef-839d-a79cae138467; Proxy: null).",
          "AccountGateResult": {
              "Status": "SKIPPED",
              "StatusReason": "Insufficient permissions to invoke AWSCloudFormationStackSetAccountGate"
          },
          "OrganizationalUnitId": ""
      },
      {
          "Account": "464691222504",
          "Region": "eu-central-1",
          "Status": "CANCELLED",
          "StatusReason": "Cancelled since failure tolerance has exceeded",
          "OrganizationalUnitId": ""
      }
  ]
  }

Screenshots N/A

Additional context N/A

Thank you very much for the support and time allocated to troubleshoot this issue!

Please, let me know shall you require any additional information.

[ericquinones]

Hi @jbernales5

I'm sorry you had this issue and thank you for the detailed report. It helped a lot with the research.

The root cause was the NoSuchBucket error:

"StatusReason": "ResourceLogicalId:StackSetConstantsCustomResourceLambda, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Error occurred while GetObject. S3 Error Code: NoSuchBucket. S3 Error Message: The specified bucket does not exist (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: a488742f-5ff5-4eef-839d-a79cae138467; Proxy: null)."

The reason you got this error was because the solution's assets need to be staged in S3 buckets in both the primary region (the region in which you are deploying the solution) and the secondary region (which you set as a CloudFormation parameter when deploying the solution's Stack).

To resolve this, you should be able to do the following after you run all the commands you listed above:

export SECONDARY_REGION=eu-central-1
aws s3 mb s3://$DIST_OUTPUT_BUCKET-$SECONDARY_REGION --region $SECONDARY_REGION
aws s3 cp ./regional-s3-assets/ s3://$DIST_OUTPUT_BUCKET-$SECONDARY_REGION/$SOLUTION_NAME/$VERSION/ --recursive --acl bucket-owner-full-control

Apologies that the instructions in the README were not clear about this. I've made a note to have it clarified when we update the solution.

[jbernales5]

Hi @ericquinones,

Happy new year, best wishes!

Thank you very much for your quick reply, I have followed your steps and I am now able to successfully deploy the stack.

Have a great day!