Changing IAM Identity Center breaks LZA

[farmerbean]

Describe the bug We have changed the delegated account for IAM IDC and subsequent running of LZA fails.

To Reproduce Change the delegated account from X -> Y AWS account

Expected behavior Nothing this difficult.

Please complete the following information about the solution:

• [ 1.3.0] Version: [e.g. v1.1.0] We've observed the following failure in the operations-X-CFN stack in the Audit account

Additionally, because the stack is in a rollback failure state, we are unable to make any further updates to LZA.

[erwaxler]

Hi @farmerbean , thank you for creating this issue! We don't recommend manually changing the delegated administrator of any AWS Service used by LZA. I believe what you are seeing is occurring because the LZA attempts to create AWS Identity Center resources within the account it believes to be the delegated admin, according to the configuration.

Based on the discussion in #72 , in v1.3.0 of the LZA, LZA is using the delegatedAdminAccount specified in the security-config.yaml account as the AWS Identity Center delegated administrator. We understand that customers may wish to specify a different account to be used for the AWS Identity Center delegated administrator, and we expect to include that feature in our next minor release.

In the meantime, I would recommend manually changing the AWS Identity Center back to the Audit account. That will enable you to continue moving forward until the next minor release, at which point you will be able to specify a different account to use as the AWS Identity Center delegated administrator.

[farmerbean]

We have run the 1.3.1 upgrade as per https://docs.aws.amazon.com/solutions/latest/landing-zone-accelerator-on-aws/update-the-solution.html

and changed the Iamconfig-IdentityCentre config as per the updated documentation https://awslabs.github.io/landing-zone-accelerator-on-aws/classes/_aws_accelerator_config.IamConfig.html#identityCenter

re-running the pipeline does not move IDC delegated admin

We have manually moved the IDC delegated admin and then re-run the pipeline. This then breaks the pipeline as per previous error

@erwaxler

[rycerrat]

Hi @farmerbean,

Thanks for reaching out and following back up. Can you confirm the following:

Which account did you manually move the IDC delegated admin to?

Does this match the delegatedAdminAccount field?

For instance, if you moved it back to Audit as per @erwaxler's recommendation, does this matchup with current delegatedAdminAccount in iam-config-yaml?

[farmerbean]

Moved the account back and fixed the cloudFormation stack ✅

Ran the 1.3.1 pipeline with the delegatedAdminAccount field set to Audit account ✅

Ran the 1.3.1 pipeline with the delegatedAdminAccount field set to sandboxIDC account ... 👎

(edited: installer stack parameters weren't updated to latest so pipeline looks to be on 1.3.0 - will update soon'ish)

[farmerbean]

Fails

❌ AWSAccelerator-OrganizationsStack-111111111111-eu-west-1 failed: Error: The stack named AWSAccelerator-OrganizationsStack-111111111111-eu-west-1 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Response object is too long.

[farmerbean]

CWL output for the Lambda

2023-03-01T23:51:04.187Z 09009961-e0a6-4139-a0ca-e065b35f68fa INFO [provider-framework] submit response to cloudformation { "Status": "FAILED", "Reason": "Delegated Admin Identity Center cannot be updated due to existing Permission Sets or Assignments. Please remove the following Permission Sets and Assignments: \n Permission Sets: [{\"Name\":\"StormPermissionSetViewOnly\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c7897568e3cee05f\",\"CreatedDate\":\"2023-02-24T09:44:23.119Z\",\"SessionDuration\":\"PT1H\"},{\"Name\":\"AdministratorAccess\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-afe45648145dffd8\",\"CreatedDate\":\"2023-02-13T20:50:28.714Z\",\"SessionDuration\":\"PT4H\"},{\"Name\":\"Billing\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-347b18bb75d7777f\",\"CreatedDate\":\"2023-02-14T12:17:01.833Z\",\"SessionDuration\":\"PT1H\"}]\n Assignments: [\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c354510bf0f3199b\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"725594e4-0061-70d6-6f08-7ac3d5fafbc4\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c354510bf0f3199b\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"82b53444-6051-7037-dda5-663a916ece5c\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-bc3f9582bc1e85c5\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"0295a414-a0a1-7025-4116-93670adcc419\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-dff164425c200c77\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"92b51484-d011-7024-6a04-94bf0335b8ba\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"421504d4-2021-7089-1ec5-752bc4282af0\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"82b53444-6051-7037-dda5-663a916ece5c\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"725594e4-0061-70d6-6f08-7ac3d5fafbc4\\\"}]\"]\n\nLogs: /aws/lambda/AWSAccelerator-Organizati-IdentityCenterAdminIdent-rUPwWhXr4TRJ\n\n Permission Sets: [{\"Name\":\"StormPermissionSetViewOnly\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c7897568e3cee05f\",\"CreatedDate\":\"2023-02-24T09:44:23.119Z\",\"SessionDuration\":\"PT1H\"},{\"Name\":\"AdministratorAccess\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-afe45648145dffd8\",\"CreatedDate\":\"2023-02-13T20:50:28.714Z\",\"SessionDuration\":\"PT4H\"},{\"Name\":\"Billing\",\"PermissionSetArn\":\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-347b18bb75d7777f\",\"CreatedDate\":\"2023-02-14T12:17:01.833Z\",\"SessionDuration\":\"PT1H\"}]\n Assignments: [\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c354510bf0f3199b\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"725594e4-0061-70d6-6f08-7ac3d5fafbc4\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-c354510bf0f3199b\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"82b53444-6051-7037-dda5-663a916ece5c\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-bc3f9582bc1e85c5\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"0295a414-a0a1-7025-4116-93670adcc419\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-dff164425c200c77\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"92b51484-d011-7024-6a04-94bf0335b8ba\\\"}]\",\"[{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"421504d4-2021-7089-1ec5-752bc4282af0\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"82b53444-6051-7037-dda5-663a916ece5c\\\"},{\\\"AccountId\\\":\\\"941111174898\\\",\\\"PermissionSetArn\\\":\\\"arn:aws:sso:::permissionSet/ssoins-6804cbf0e09ea22e/ps-208ec9669212112a\\\",\\\"PrincipalType\\\":\\\"GROUP\\\",\\\"PrincipalId\\\":\\\"725594e4-0061-70d6-6f08-7ac3d5fafbc4\\\"}]\"]\n at lm (/var/task/index.js:32:17746)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async um (/var/task/index.js:32:17338)\n at async Runtime.sm [as handler] (/var/task/index.js:32:15969)", "StackId": "arn:aws:cloudformation:eu-west-1:885614340325:stack/AWSAccelerator-OrganizationsStack-885614340325-eu-west-1/52d95f20-aee3-11ed-bb42-0a9840e9e66b", "RequestId": "45bc73c3-8c11-44a5-b93b-7fa1b7091a07", "PhysicalResourceId": "0898c874-a5db-49c6-a5e0-f875016e9e22", "LogicalResourceId": "IdentityCenterAdminidentityCenterAdminEB714AB1" }

[rycerrat]

Hi @farmerbean,

Looks like there is an issue with returning the error response which includes all of the PermissionSets and Assignments (I'll open up a bug request to reduce verbosity on this). However, if you manually remove these Permission Sets and Assignments, this should fix the issue. This behavior is intentional as we want to ensure that dangling Permission Sets and Assignments are not left in the previous Delegated Admin Account after it has been updated.

Please let me know if you continue running into issues.

Cheers, Ryan

[farmerbean]

Hi ryan - I have removed the permission set we created with LZA in audit account.

The other permission sets were created in the main account and not in Audit account.

To be clear: I simply need to remove all permission sets from the source account (Audit) and this will "definitely fix" the issue?

[farmerbean]

Both AdministratorAccess and Billing permissionSets only exist in the Main Billing Account ..

[sigfigsteve]

I'm having the same issue - brown field where I already have identity center set up in the Management account and I'm trying to move the permissionsets and assignments to LZA. After setting delegatedAdminAccount under the identityCenter configs, I get:

AWSAccelerator-OrganizationsStack-854Management-us-east-2 failed: Error: The stack named AWSAccelerator-OrganizationsStack-854661928400-us-east-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Response object is too long.

This happens in the "Organization stage" I'm using a config:

identityCenter:
  name: identityCenter1
  delegatedAdminAccount: Management
  identityCenterPermissionSets:
    - name: Admin
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/AdministratorAccess
      sessionDuration: 720
  identityCenterAssignments:
    - name: Admin
      permissionSetName: Admin
      principalId: 'ad admin group' 
      principalType: GROUP
      deploymentTargets:
        organizationalUnits:
          - Root

This is a new permissionset that doesn't conflict with existing ones. Prior to setting delegatedAdminAccount, I was getting:

AWSAccelerator-OperationsStack-582Audit-us-east-2 failed: Error: The stack named AWSAccelerator-OperationsStack-582Audit-us-east-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "User: arn:aws:sts::582Audit:assumed-role/cdk-accel-cfn-exec-role-582Audit-us-east-2/AWSCloudFormation is not authorized to perform: sso:CreateAccountAssignment on resource: arn:aws:sso:::account/854Master (Service: SsoAdmin, Status Code: 400, Request ID: b61da444-...)" (RequestToken: df02454e-..., HandlerErrorCode: GeneralServiceException)

This error happened in the "Operations" task of the "Deploy" stage. My centralSecurityServices delegatedAdminAccount is Audit. There is no identitycenter in my audit account.

[farmerbean]

We tried this a bunch of ways @sigfigsteve with different accounts but the long and short is that if you do "any" assignments with identity centre (including your ones above), you must use Audit account (line 3 of your snippet)

[sigfigsteve]

I'm getting the same permissions error even when I set the delegatedAdminAccount to Audit. The permission sets are created successfully, but the sso:CreateAccountAssignment fails when the audit account tries to create an assignment on the management account.

[zakbatinica-reply]

I'm getting the same permissions error even when I set the delegatedAdminAccount to Audit. The permission sets are created successfully, but the sso:CreateAccountAssignment fails when the audit account tries to create an assignment on the management account.

Just in case it helps anyone else, this is actually a different issue and a limitation of the delegated admin as it can't manage permission sets for the management account (presumably, to avoid escalating permissions). Full details: https://docs.aws.amazon.com/singlesignon/latest/userguide/delegated-admin.html

[frankscalzo]

I'm getting the same permissions error even when I set the delegatedAdminAccount to Audit. The permission sets are created successfully, but the sso:CreateAccountAssignment fails when the audit account tries to create an assignment on the management account.

@sigfigsteve as @zakbatinica-reply stated there are limitations to deploying permission sets from the delegated account will not allow you to put them on the Main account that SSSO resides from, there are 3 ways around this 1 you can deploy at Root OU and exclude the Management account 2. you can deploy to a specific account only or 3. you can deploy to a specific OU above root IF you have nested OU you must list them as well it will not matriculate down the OU structure SEE example code :

`identityCenter: name: MYSSO delegatedAdminAccount: Audit identityCenterPermissionSets:

• name: AWS-GBL-Sec-Admin policies: awsManaged:
  • arn:aws:iam::aws:policy/SecurityAudit
  • arn:aws:iam::aws:policy/job-function/SystemAdministrator customerManaged: [] sessionDuration: 60
• name: AWS-GLB-Contributor policies: awsManaged:
  • arn:aws:iam::aws:policy/PowerUserAccess
  • arn:aws:iam::aws:policy/job-function/SystemAdministrator customerManaged: [] sessionDuration: 60
• name: AWS-GLB-ReadOnly policies: awsManaged:
  • arn:aws:iam::aws:policy/ReadOnlyAccess customerManaged: [] sessionDuration: 60
• name: AWS-Audit-ReadOnly policies: awsManaged:
  • arn:aws:iam::aws:policy/ReadOnlyAccess customerManaged: [] sessionDuration: 60

• name: AWS-EpicProd-Contributor policies: awsManaged:

  • arn:aws:iam::aws:policy/PowerUserAccess
  • arn:aws:iam::aws:policy/job-function/SystemAdministrator customerManaged: [] sessionDuration: 60

  identityCenterAssignments:

  • name: AWS-GBL-Sec-Admin permissionSetName: AWS-GBL-Sec-Admin principalId: 85c1d77340f4-10c1-70e0-75db-a2e41a9ac6e6 principalType: GROUP deploymentTargets: organizationalUnits:
    • Root excludedAccounts:
    • Management
  • name: AWS-GLB-Contributor permissionSetName: AWS-GLB-Contributor principalId: 85c1d77340f4-a0b1-7039-7980-85c1d77340f4 principalType: GROUP deploymentTargets: organizationalUnits:
    • Root excludedAccounts:
    • Management
  • name: AWS-GLB-ReadOnly permissionSetName: AWS-GLB-ReadOnly principalId: 85c1d77340f4-90e1-70e6-7c89-0b179d9fc29e principalType: GROUP deploymentTargets: organizationalUnits:
    • Root excludedAccounts:
    • Management
  • name: AWS-Audit-ReadOnly permissionSetName: AWS-Audit-ReadOnly principalId: 85c1d77340f4-c031-70e3-c6e1-6f880fc55a73 principalType: GROUP deploymentTargets: accounts:
    • Audit
  • name: AWS-EpicProd-Contributor permissionSetName: AWS-EpicProd-Contributor principalId: e4c8a448-00b1-70ec-c3a9-85c1d77340f4 principalType: GROUP deploymentTargets: organizationalUnits:
    • HIS`

[sigfigsteve]

Thank you @frankscalzo - excluding the Management account was the missing piece to make this work for me. I ended up using customizations-config to add a few permissions to the management account and identityCenterAssignments for everything else. Now if I add new workload accounts, they're automatically configured with the appropriate permissionSets.

[rgd11]

Please see https://awslabs.github.io/landing-zone-accelerator-on-aws/classes/_aws_accelerator_config.IdentityCenterConfig.html#delegatedAdminAccount for more information regarding Delegated Admin Accounts and Identity Center configuration with LZA.

Closing this issue as the thread topic has changed and workarounds / usage has been also discussed.

We are internally tracking the request for configuring permission sets for the management account