awslabs / landing-zone-accelerator-on-aws

Deploy a multi-account cloud foundation to support highly-regulated workloads and complex compliance requirements.
https://aws.amazon.com/solutions/implementations/landing-zone-accelerator-on-aws/
Apache License 2.0
563 stars 448 forks source link

Logging stack is trying to assume the `managementAccountAccessRole` for deployments within the Management account #603

Open kisst opened 1 month ago

kisst commented 1 month ago

Describe the bug

❌ Deployment failed: AccessDenied: User: arn:aws:sts::<managmentaccountid>:assumed-role/AWSAccelerator-PipelineSt-AdminCdkToolkitRole292E16-UniqRandomStr/AWSCodeBuild-ec87b680-634e-4429-9aa5-252d3350e8a9 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<managmentaccountid>:role/OrganizationAccountAccessRole
    at Request.extractError (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/codebuild/output/src331/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'AccessDenied',
  time: 2024-10-11T03:53:55.657Z,
  requestId: '1243baf1-01f5-45fe-b748-a2d27c88cae4',
  statusCode: 403,
  retryable: false,
  retryDelay: 96.37077033090154
}
2024-10-11 03:53:55.659 | error | toolkit | User: arn:aws:sts::<managmentaccountid>:assumed-role/AWSAccelerator-PipelineSt-AdminCdkToolkitRole292E16-UniqRandomStr/AWSCodeBuild-ec87b680-634e-4429-9aa5-252d3350e8a9 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<managmentaccountid>:role/OrganizationAccountAccessRole
Deployment failed
error Command failed with exit code 1.

As per the above error message, for some reason the deployment of the Logging stack running within the management account itself, is trying to use the managementAccountAccessRole within the account

To Reproduce Unsure what's changed, rebuild it with an older version of the config zip ( one which run before ) and it failed this time, no changes in the account, or in the config.

Please complete the following information about the solution:

Additional context After a full pipeline run, made a single line change in iam, tried to deploy it, but it failed on logging, so after reading the logs, I tried rolling back the config change, and rebuild, but it still failed.

JimToupet commented 1 month ago

Are you using Control Tower ?

If so, you have to set this in global-config.yaml

managementAccountAccessRole: AWSControlTowerExecution # UPDATE: If using Control Tower, set to AWSControlTowerExecution

And set the useManagementAccessRole to true:

cdkOptions:
  useManagementAccessRole: true
  customDeploymentRole: AWSAccelerator-PipelineRole # UPDATE: If using a custom AcceleratorPrefix in the installer stack, this need to be updated accordingly
kisst commented 1 month ago

No Control Tower, in use in this install in the meantime I opened a support ticket within the effected AWS account as well, where the workaround suggested by the support was to create this Org role in the Master account too by hand while the fix is being worked on, as apparently this error is observed in other installations too..

schnipseljagd commented 2 weeks ago

We are running into a similar issue. We've got customizations cloudformation stacks with target:

      deploymentTargets:
        organizationalUnits:
          - Root

which deploys the stack to all member accounts + the management account.

So far it behaved correctly and did assume-role for the member accounts and for the management account assume-role was skipped.

However somehow for the latest pipeline run LZA tries to do an assume role on the AWSControlTowerExecution role when deploying to the management account.

The interesting thing is that we haven't updated the LZA version.

This is the diff of the change that triggered the new behavior which seems rather unrelated:

diff --git a/security-config.yaml b/security-config.yaml
index a0bf456..a07939c 100644
--- a/security-config.yaml
+++ b/security-config.yaml
@@ -26,38 +26,8 @@ centralSecurityServices:
       destinationType: S3
       exportFrequency: FIFTEEN_MINUTES
   securityHub:
-    enable: true
-    regionAggregation: true
-    excludeRegions: []
-    standards:
-      # https://docs.aws.amazon.com/securityhub/latest/userguide/fsbp-standard.html
-      - name: AWS Foundational Security Best Practices v1.0.0
-        deploymentTargets:
-          organizationalUnits:
-            - Root
-        enable: true
-        controlsToDisable:
-          - IAM.6
-          - Inspector.1
-          - Inspector.2
-          - Inspector.3
-          - Inspector.4
-          - GuardDuty.1
-          - GuardDuty.5
-          - GuardDuty.6
-          - GuardDuty.8
-          - GuardDuty.9
-          - GuardDuty.10
-          - Macie.1
-      - name: CIS AWS Foundations Benchmark v1.2.0
-        deploymentTargets:
-          organizationalUnits:
-            - Root
-        enable: false
+    enable: false
+    standards: []
   ssmAutomation:
     excludeRegions: []
     documentSets: []