Bootstrap stage fails with SCP implicit deny (v1.7.0)

[richardkeit]

Describe the bug Bootstrap fails with:

User: arn:aws:sts::XXXX:assumed-role/AWSControlTowerExecution/acceleratorBootstrapCheck is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:ap-southeast-2:XXXX:parameter/cdk-bootstrap/accel/version because no service control policy allows the ssm:GetParameter action

To Reproduce Upgrade to 1.7.0

Expected behavior Bootstrap doesnt fail

Please complete the following information about the solution:

• [x] Version: v1.7.0

• [x] Region: ap-southeast-2

• [ ] Was the solution modified from the version published on this repository?

• [ ] If the answer to the previous question was yes, are the changes available on GitHub?

• [ ] Have you checked your service quotas for the services this solution uses?

• [ ] Were there any errors in the CloudWatch Logs?

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context Debugging found the custom resource remove the AWSFullAccess which is required at each OU:

{
        "eventVersion": "1.09",
        "userIdentity": {
            "type": "AssumedRole",
            "principalId": "YYYYYYY:AWSAccelerator-AccountsSt-CustomOrganizationsAttac-cxRdNSFHLylv",
            "arn": "arn:aws:sts::XXXXXXXX:assumed-role/AWSAccelerator-AccountsSt-CustomOrganizationsAttach-jBrA230Tcs9U/AWSAccelerator-AccountsSt-CustomOrganizationsAttac-cxRdNSFHLylv",
            "accountId": "XXXXXXXX",
            "accessKeyId": "ASIAVGPSZPHQAQ3KMMU2",
            "sessionContext": {
                "sessionIssuer": {
                    "type": "Role",
                    "principalId": "YYYYYYY",
                    "arn": "arn:aws:iam::XXXXXXXX:role/AWSAccelerator-AccountsSt-CustomOrganizationsAttach-jBrA230Tcs9U",
                    "accountId": "XXXXXXXX",
                    "userName": "AWSAccelerator-AccountsSt-CustomOrganizationsAttach-jBrA230Tcs9U"
                },
                "attributes": {
                    "creationDate": "2024-06-04T02:51:46Z",
                    "mfaAuthenticated": "false"
                }
            }
        },
        "eventTime": "2024-06-04T02:53:06Z",
        "eventSource": "organizations.amazonaws.com",
        "eventName": "DetachPolicy",
        "awsRegion": "us-east-1",
        "sourceIPAddress": "34.231.109.134",
        "userAgent": "aws-sdk-js/3.410.0 ua/2.0 os/linux#5.10.216-225.855.amzn2.x86_64 lang/js md/nodejs#18.20.0 api/organizations#3.410.0 exec-env/AWS_Lambda_nodejs18.x AwsSolution/SO0199/1.6.3",
        "requestParameters": {
            "targetId": "ou-hb6b-3ayeuibv",
            "policyId": "p-FullAWSAccess"
        },
        "responseElements": null,
        "requestID": "6fb11844-0623-43ce-8f7d-210e1473ee90",
        "eventID": "894e87e5-5e18-4643-bd5d-45c08e3137f9",
        "readOnly": false,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": "XXXXXXXX",
        "eventCategory": "Management",
        "tlsDetails": {
            "tlsVersion": "TLSv1.3",
            "cipherSuite": "TLS_AES_128_GCM_SHA256",
            "clientProvidedHostHeader": "organizations.us-east-1.amazonaws.com"
        }
    }

[dfevre]

👍

[erwaxler]

Hey @richardkeit , thanks for writing this up. I know you mentioned in the linked issue that you narrowed this down to not setting the strategy for the SCP explicitly. I'm reviewing the code, and still am unsure of why that would be occurring.

Specifically I'm looking at this code:

      // if SCP strategy is allow-list, then FullAWSAccess policy should be detached
      if (strategy === 'allow-list' && fullAwsAccessPolicyAttached) {
        console.log('detaching FullAWSAccess policy because the strategy is allow-list');
        await detachSpecificPolicy(organizationsClient, 'p-FullAWSAccess', targetId);
      }

      // if SCP strategy is changed from allow-list to deny list, then FullAWSAccess policy should be attached
      if (strategy === 'deny-list' && !fullAwsAccessPolicyAttached) {
        console.log('attaching FullAWSAccess policy because the strategy is deny-list');
        await attachSpecificPolicy(organizationsClient, 'p-FullAWSAccess', targetId);
      }

This is where I would expect the bug to exist. Can you share the logs of the AWSAccelerator-AccountsSt-CustomOrganizationsAttac-cxRdNSFHLylv Lambda function? They may provide more insight on what is leading to the detachment of that policy.