IAM Identity Center (ICC) (LZA v1.7.0+) fails to successfully create Management Account assignments - Failure message mentions missing iam:CreateSAMLProvider permission on the IIC Assignment Lambda IAM Role

CirrusHQ-Pipeline-User commented 6 days ago

Please provide the following information: -->

Describe the bug

(NOTE WE HAVE COVERED POTENTIALLY SENSITIVE INFO WHERE APPLICABLE WITH X's)

We have an IAM Identity Center Instance which we've configured an external IdP for and is controlled by LZA. We deployed permission sets and assignments to many OUs and accounts, including the Management Account via LZA, which went through successfully without issue.

However, we noticed that all assignments except for the Management Account were successfully created. I investigated the CloudTrail logs for the CreateAccountAssignment API call and found one showing that the assignment for the Management Account was attempted and was recorded as IN_PROGRESS. Using the following command:

aws sso-admin describe-account-assignment-creation-status --account-assignment-creation-request-id xxxxxxxxx --instance-arn arn:aws:sso:::instance/ssoins-xxxxxxxxx

I was able to discover the error message, which I will paste below:

"FailureReason": "Received a 403 status error: Access denied by IAM. Please check your policy, or wait for role propagation to complete. IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource: arn:aws:iam::xxx:saml-provider/AWSSSO_xxx_DO_NOT_DELETE because no identity-based policy allows the iam:CreateSAMLProvider action (Service: AmazonIdentityManagement; Status Code: 403; Error Code: AccessDenied; Request ID: xxx; Proxy: null)"

This is a surprising error, and the fact it didn't appear in the other assignments shows it's a Management Account assignment-specific issue. @erwaxler mentioned on a separate but linked thread: https://github.com/awslabs/landing-zone-accelerator-on-aws/issues/215

https://docs.aws.amazon.com/singlesignon/latest/userguide/iam-auth-access-using-id-policies.html#policyexamplemanageconnecteddirectory

However, the permission in the error code (iam:CreateSAMLProvider) is not mentioned on that page.

Since the IAM Role is managed by LZA, we are unsure how to proceed. This seems like it will affect other customers, with an unclear path on how to proceed except for manually configuring the assignment for the Management Account, which defeats the purpose of the feature being added in v1.7.0, allowing it to be assigned by LZA.

To Reproduce

Enable IAM Identity Centre
Configure an external IdP (such as AzureAD)
Deploy IAM Identity Centre configurations, such as permission sets which are assigned, by LZA, to the Management Account
Use CloudTrail to investigate the request ID of the failed assignment attempt to the Management Account
Use the aws sso-admin describe-account-assignment-creation-status to check for failure codes of the failed assignment attempt to the Management Account

Expected behavior Assignments to the Management Account are successful through LZA

Please complete the following information about the solution:

[ ] Version: v1.7.1
[ ] Region: eu-west-2
[ ] Was the solution modified from the version published on this repository? No
[ ] If the answer to the previous question was yes, are the changes available on GitHub? N/a
[ ] Have you checked your service quotas for the services this solution uses? N/a
[ ] Were there any errors in the CloudWatch Logs? Yes and error code was pasted above

Screenshots N/a

Additional context Linked issue is https://github.com/awslabs/landing-zone-accelerator-on-aws/issues/215

erwaxler commented 6 days ago

Hi @CirrusHQ-Pipeline-User , thanks for creating this issue. I've created an item in the team's backlog to track this work, I will update this ticket accordingly. Thank you for your support of the Landing Zone Accelerator!

hemanth-m19 commented 5 days ago

Hello! @CirrusHQ-Pipeline-User and @erwaxler, I wanted to share some insights on our configuration that might help in diagnosing this issue effectively.

We've recently updated our LZA version to the latest (v1.7.1). Additionally, we have our IAM Identity Centre configured with an external identity provider (Okta) as well. This version update has successfully resolved previous issues, and as of now, we haven't encountered the problem you mentioned here

Below is the sample config I tested just then to double check:

identityCenter:
  name: IdentityCenter
  delegatedAdminAccount: Audit
  identityCenterPermissionSets:
    - name: OrgViewOnlyAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      sessionDuration: 60

    - name: OrgReadOnlyAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/ReadOnlyAccess
        inlinePolicy: iam-policies/OrgReadOnlyAccess-inline-policy.json
      sessionDuration: 60

  identityCenterAssignments:
  - name: Assignment1
    permissionSetName: OrgViewOnlyAccess
    principals:
      - type: GROUP
        name: OrgAWSViewers
    deploymentTargets:
      organizationalUnits:
        - Root # Sucessfully assigned to all accounts

  - name: Assignment2
    permissionSetName: OrgReadOnlyAccess
    principals:
      - type: GROUP
        name: OrgAWSReaders
    deploymentTargets:
      accounts:
        - Management # Sucessfully assigned to Management account

CirrusHQ-Pipeline-User commented 5 days ago

Hi @hemanth-m19 (CC: @erwaxler),

Thank you for the additional context and sample Config.

What's weird is that I seem to have a similiar Config to how you've set it up and I wouldn't have suspected it to be a Config issue anyway since validation tests passed and the Pipeline successfully completed.

I will paste my Config below:

providers: []
policySets: []
roleSets: []
groupSets: []
userSets: []

identityCenter:
  name: identityCenter
  delegatedAdminAccount: SharedServices
  identityCenterPermissionSets:
    - name: ReadOnlyAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      sessionDuration: 480

    - name: AdministratorAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/AdministratorAccess
      sessionDuration: 480

    - name: FinanceAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/AWSSupportAccess
          - arn:aws:iam::aws:policy/job-function/Billing                 
      sessionDuration: 480

    - name: DeveloperAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/PowerUserAccess
      sessionDuration: 480

    - name: Monitoring
      policies:
        inlinePolicy: iam-policies/monitoring-policy.json
      sessionDuration: 480

  identityCenterAssignments:
  #adjust the below so that they are assigned to the correct OU/Account
    - name: AdministratorAccess
      permissionSetName: AdministratorAccess
      principals:
        - type: GROUP
          name: AWS-AdministratorAccess
      deploymentTargets:
        organizationalUnits:
          - Workload
          - Infrastructure
          - Security
          - Quarantine
        accounts:
          - Management # did not get assigned as recieved the IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource 

    - name: Read-Only
      permissionSetName: ReadOnlyAccess
      principals:
        - type: GROUP
          name: AWS-ReadOnlyAccess
      deploymentTargets:
        organizationalUnits:
          - Workload
          - Infrastructure
          - Security
          - Quarantine
        accounts:
          - Management # did not get assigned as recieved the IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource 

    - name: Finance
      permissionSetName: FinanceAccess
      principals:
        - type: GROUP
          name: AWS-BillingAccess
      deploymentTargets:
        organizationalUnits:
          - Workload
        accounts:
          - Management # did not get assigned as recieved the IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource 

    - name: Developer
      permissionSetName: DeveloperAccess
      principals:
        - type: GROUP
          name: AWS-DeveloperAccess
      deploymentTargets:
        organizationalUnits:
          - Workload
        accounts:
          - Management # did not get assigned as recieved the IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource 

    - name: Monitoring
      permissionSetName: Monitoring
      principals:
        - type: GROUP
          name: AWS-MonitoringAccess
      deploymentTargets:
        organizationalUnits:
          - Workload
        accounts:
          - Management # did not get assigned as recieved the IAM Error: User: arn:aws:sts::xxx:assumed-role/xxxx-IdentityCenterStac-CustomIdentityCenterAssig-xxx/xxx-IdentityCenterStac-CustomIdentityCenterAssi-xxx is not authorized to perform: iam:CreateSAMLProvider on resource

Again, all that was done was the steps I have listed in recreate, the only difference between our setup and yours from what you mention is that we use AzureAD instead of Okta (but that should be no issue). With that in mind I believe this still must be a bug or managed LZA permissions issue as there is no other pain point we can think of which would be causing this.

Interested to hear your thoughts.

hemanth-m19 commented 1 day ago

@CirrusHQ-Pipeline-User

It's interesting that an error is arising stating the IAM role isn't authorized to perform the action iam:CreateSAMLProvider.

From my understanding, when adding assignments to a Management account configured with an external provider, the IAM role only needs the iam:GetSAMLProvider permission (although there might be variations; adding this only permission to IAM role has been functional in another internal assignments pipeline of mine).

Additionally, this permission is already in place for the Custom::IdentityCenterAssignments Lambda function. You can refer to this link for more details.

CirrusHQ-Pipeline-User commented 1 hour ago

@hemanth-m19 & @erwaxler - PLEASE READ THE FULL RESPONSE BELOW, I THINK I'VE FOUND THE PROBLEM AND POTENTIAL FIX AROUND THIS.

Thanks for your response and clarification. I followed a similar process with another separate AWS Organization and encountered the same issue. Here are the steps I followed:

Enable IAM Identity Centre in the Management Account (to create the Org IIC Instance)
Deploy LZA iam-config.yml with NO assignments and NO permission sets, just to set the delegated admin to the SharedServices Account
I then entered the SharedServices Account to setup Azure AD as an Identity Provider which thus allowed my Azure AD Team to send over Groups and Users into IIC
I then deployed iam-config.yml through LZA with assignments and permission sets

The outcome of this was the same as the previous Org, ALL assignments worked EXCEPT for the Management Account ones. For this new Org we also upgraded version v1.6.3 to v1.7.1 and are getting the exact same error (when running aws sso-admin describe-account-assignment-creation-status) which I will paste below.

Please note, again, ALL assignments worked except for the one to the Management Account, despite including the Management Account via deploymentTargets > organizationalUnits > Root in one assignment and trying via deploymentTargets > accounts > Management in another.

{
    "AccountAssignmentCreationStatus": {
        "CreatedDate": "2024-07-03T10:12:39.038000+00:00",
        "FailureReason": "Received a 403 status error: Access denied by IAM. Please check your policy, or wait for role propagation to complete. IAM Error: User: arn:aws:sts::xxxx:assumed-role/xxxx-IdentityCenterSta-CustomIdentityCenterAssg-xxxx/xxxx-IdentityCenterSta-CustomIdentityCenterAssi-xxxx is not authorized to perform: iam:CreateSAMLProvider on resource: arn:aws:iam::xxxx:saml-provider/AWSSSO_xxxx_DO_NOT_DELETE because no identity-based plicy allows the iam:CreateSAMLProvider action (Service: AmazonIdentityManagement; Status Code: 403; Error Code: AccessDenied; Request ID: xxxx; Proxy: null)",
        "PermissionSetArn": "arn:aws:sso:::permissionSet/ssoins-xxxx/xxxx",
        "PrincipalId": "xxxx",
        "PrincipalType": "GROUP",
        "RequestId": "xxxx",
        "Status": "FAILED",
        "TargetId": "xxxx",
        "TargetType": "AWS_ACCOUNT"
    }
}

Here is the iam-config.yml which was used:

providers: []
policySets: []
roleSets: []
groupSets: []
userSets: []

identityCenter:
  name: identityCenter
  delegatedAdminAccount: SharedServices
  identityCenterPermissionSets: 

    - name: ReadOnlyAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/ReadOnlyAccess
      sessionDuration: 480

    - name: AdministratorAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/AdministratorAccess
      sessionDuration: 480

    - name: FinanceAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/job-function/Billing                 
      sessionDuration: 480

    - name: DeveloperAccess
      policies:
        awsManaged:
          - arn:aws:iam::aws:policy/job-function/SystemAdministrator
      sessionDuration: 480

  identityCenterAssignments: 
#adjust the below so that they are assigned to the correct OU/Account
    - name: AWS-AdminAccessEverything
      permissionSetName: AdministratorAccess
      principals:
        - type: GROUP
          name: AWS-AdminAccessEverything
      deploymentTargets:
        organizationalUnits:
          - Root

    - name: AWS-ReadOnlyAccessEverything
      permissionSetName: ReadOnlyAccess
      principals:
        - type: GROUP
          name: AWS-ReadOnlyAccessEverything
      deploymentTargets:
        organizationalUnits:
          - Root

    - name: AWS-ReadOnlySecurity
      permissionSetName: ReadOnlyAccess
      principals:
        - type: GROUP
          name: AWS-ReadOnlySecurity
      deploymentTargets:
        organizationalUnits:
          - Security

    - name: AWS-ReadOnlyIAM
      permissionSetName: ReadOnlyAccess
      principals:
        - type: GROUP
          name: AWS-ReadOnlyIAM
      deploymentTargets:
        organizationalUnits:
          - Infrastructure

    - name: AWS-AdminManagementOnly
      permissionSetName: AdministratorAccess
      principals:
        - type: GROUP
          name: AWS-AdminManagement
      deploymentTargets:
        accounts:
          - Management

All assignments worked except for the one to the Management Account, despite including the Management Account via deploymentTargets > organizationalUnits > Root in one assignment and trying via deploymentTargets > accounts > Management in another.

After investigating, I found that when creating an external Identity Provider in IAM Identity Center, it also creates an Identity Provider within the IAM Console. It appears SSO tries to create an Identity Provider from the Management Account when creating Management Account Assignments. Since it is not available (as the external Identity Provider integration was created from my SharedServices Account, the delegated Admin of IIC as per best practices), the process fails unless the Custom IIC Assignments resource in the Management Account has the correct permissions, and currently it does not to support the above workflow.

The new update (v1.7.0) deploys IIC-related resources in the Management Account (as noted in the v1.7.0 release notes), and it needs to use the SAML Provider which is not available within the Management Account, if the external Identity Provider is first created from the IIC Delegated Admin Account

Here are screenshots from the IAM Console in the Identity Providers section:

SharedServices Account: IdP-Shared-Services

Management Account: IdP-Management

To fix this, it appears the Management Account also needs a SAML Identity Provider to create assignments. I recommend the LZA Team test this process and include the necessary permissions in a future release, as once the permission for iam:CreateSAMLProvider is added, more may still be needed so a full replication/investigation is required from the LZA Team.

Additionally, clarifying the process of using Delegated Admins vs. Management Account for IIC could prevent future customers from encountering this issue.

Thanks and I look forward to your feedback.

Best,

awslabs / landing-zone-accelerator-on-aws

IAM Identity Center (ICC) (LZA v1.7.0+) fails to successfully create Management Account assignments - Failure message mentions missing iam:CreateSAMLProvider permission on the IIC Assignment Lambda IAM Role #496