Unable to import an account that's had it's email address changed

[daveshepherd]

AFT Version: 1.10.3

Terraform Version & Provider Versions Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version

Terraform v1.5.1
on darwin_amd64
+ provider registry.terraform.io/hashicorp/archive v2.4.0
+ provider registry.terraform.io/hashicorp/aws v4.67.0
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/hashicorp/time v0.9.1

Your version of Terraform is out of date! The latest version
is 1.5.3. You can update by downloading from https://www.terraform.io/downloads.html

terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] ~> 4.27
└── module.atf
    ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0, < 5.0.0
    ├── provider[registry.terraform.io/hashicorp/local]
    ├── module.aft_code_repositories
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/local]
    ├── module.aft_iam_roles
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.ct_management_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.log_archive_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.log_archive_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.aft_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.aft_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.audit_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── module.audit_service_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── module.ct_management_exec_role
    │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_ssm_parameters
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/random]
    ├── module.aft_backend
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_feature_options
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    ├── module.aft_lambda_layer
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   ├── provider[registry.terraform.io/hashicorp/random]
    │   └── provider[registry.terraform.io/hashicorp/local]
    ├── module.aft_account_provisioning_framework
    │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0
    ├── module.aft_account_request_framework
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0
    │   └── provider[registry.terraform.io/hashicorp/time]
    ├── module.aft_customizations
    │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
    │   └── provider[registry.terraform.io/hashicorp/local]
    └── module.packaging
        └── provider[registry.terraform.io/hashicorp/archive]

Providers required by state:

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/hashicorp/archive]

    provider[registry.terraform.io/hashicorp/local]

    provider[registry.terraform.io/hashicorp/random]

    provider[registry.terraform.io/hashicorp/time]

Bug Description I have an account I'm trying to import into account factory for terraform, so that we can ensure global/account customisations can be applied. However, this account has had it's name and email address changed, so AFT can't find it in the service catalog and tries to provision a new account. The provisioning fails as there is already an account with that email address.

To Reproduce Steps to reproduce the behavior:

1. Create an account via account factory in control tower (not via AFT), use an email for the account email e.g. old_email@example.com
2. Once that account is provisioned, recover root credentials and change the account email to something else. e.g. new_email@example.com. Following these instructions: https://docs.aws.amazon.com/controltower/latest/userguide/change-account-email.html - note that the step for updating the provisioned product in the service catalog involves providing the email address that was originally used to provision the account (i.e. old_email@example.com).

3. Add the account to the AFT aft-account-request git repo using the new email address:

   module "example_account" {
   source = "./modules/aft-account-request"
   
   control_tower_parameters = {
   AccountEmail              = "new_email@example.com"
   AccountName               = "Example Account
   ManagedOrganizationalUnit = "Sandbox
   SSOUserEmail              = "new_email@example.com"
   SSOUserFirstName          = "New"
   SSOUserLastName           = "Email"
   }
   change_management_parameters = {
   change_requested_by = "Dave"
   change_reason       = "importing existing account into AFT"
   }
   account_customizations_name = "example-customisations"
   }

4. Let AFT do it's thing, and you get a notification on the SNS topic:

   An error occurred in the 'aft-account-request-processor' Lambda function.
   For more information, search AWS Request ID '2f2fcae2-24f7-407f-af70-1febaa925ad6' in CloudWatch log group '/aws/lambda/aft-account-request-processor'
   Error Message: CT Request is not valid

Expected behavior

Any help on how to progress with getting this account into AFT would be appreciated.

Ideally, either AFT should handle the situation where the service catalog email no longer matches the account email, or there should be clear instructions on how to resolve this scenario.

Related Logs You can see in the cloudwatch logs for aft-account-request-action-trigger:

• Did not find account with matching email in healthy status in Account Factory
• New account request received

I couldn't see much information in the aft-account-request-processor logs, other than the error posted to the SNS topic.

Additional context I also tried adding the request to aft-account-request using the original email (e.g. old_email@example.com); AFT recognised it as an existing account but then the aft-account-request-action-trigger errored:

Error Message: Account email old_email@example.com not found in Organization

[anthony-esper-allspring]

Hi @daveshepherd

I had to adjust many accounts in AFT after their emails were changed outside AFT but were already managed by AFT. To do this I had to update both the account_request module stanza in TF and the dynamodb table entries for those accounts.

Now I am in your boat. I have an account that is not managed by AFT and I need to import it to have it under AFT management. There is no customization pipeline and the account is not inside of the aft-request-metadata table. Thank you for the clue - I found the same exact error in the request-processor logs.

{ "time_stamp": "2023-07-25 17:43:58,480", "module": "aft_account_request_processor", "log_level": "ERROR", "log_message": { "FILE": "aft_account_request_processor.py", "METHOD": "lambda_handler", "EXCEPTION": "CT Request is not valid" } }

Traceback (most recent call last): File "/var/task/aft_account_request_processor.py", line 118, in lambda_handler raise RuntimeError("CT Request is not valid") RuntimeError: CT Request is not valid

I found that the account request is indeed inside of the aft-request and aft-request-audit table.

I checked the email in Control Tower under Accounts and see that the email is now correct.

**What was really strange when I originally checked the the old email was listed in the outside account list table (/controltower/home/dashboard?region=us-east-1#managed-accounts-table) but when clicking inside the account on the GUI it showed the new email. I clicked 'update account' on that account per AWS support and it then updated the email on the outside managed accounts table. I am pondering what is the story here.

I have not tested the provisioning step again after "Update Account"ing the account from Control Tower. Also, I'm wondering if I need to remove the entries from the dynamodb tables again before I retest via a redeploy of Account Request module.

Please keep me posted and I will do the same.

[anthony-esper-allspring]

Deeper Digging

From AFT_COMMON

The handle_customization_request() method in the AccountRequestRecordHandler class is responsible for processing customization requests for AWS accounts. When an account request is received, this method is called to trigger the customization process for the account. It prepares the necessary data and invokes an AWS Step Function to handle the customization workflow.

Let's break down how handle_customization_request() works:

Get Account Information: The method retrieves the account request data from the DynamoDB record. It also obtains the account ID associated with the account request. The account ID is fetched using an OrganizationsAgent instance, which interacts with AWS Organizations to retrieve the account ID based on the account request email.

Build Account Customization Payload: The method builds the payload that will be used to customize the AWS account. The payload includes information about the account request, the associated AWS account, and any relevant data needed for customization. It is structured as an AftInvokeAccountCustomizationPayload.

Retrieve Account Provisioning Event: Before proceeding with customization, the method needs an event to represent the provisioning of the account. This event is constructed based on the account request data and is used to track the status of the account provisioning process.

The provisioning event is obtained by calling the build_aft_account_provisioning_framework_event() function, which constructs the event using the account request information.

Invoke AWS Step Function: With the account customization payload and the provisioning event ready, the method proceeds to invoke an AWS Step Function. The Step Function represents the workflow for account customization. The name of the Step Function is fetched from an SSM (Systems Manager) parameter.

The Step Function is triggered with the prepared payload as its input. The Step Function execution will then follow the defined workflow, which may include multiple steps and state transitions, until the customization process is completed.

[anthony-esper-allspring]

To verify that modify_ct_request_is_valid() is true, you need to check the following:

Existing Account Parameters: Retrieve the existing Control Tower parameters for the account in question. These parameters are typically stored in some data store or database that holds the account information.

New Account Parameters: Obtain the new Control Tower parameters that are part of the modification request.

Ensure Specific Parameters Can Be Modified: Check if the modification request only includes parameters that are allowed to be modified. In the provided modify_ct_request_is_valid() function, it iterates through the old_ct_parameters (existing parameters) and new_ct_parameters (new parameters) dictionaries and compares each parameter's value.

The function allows modification if the following conditions are met:

The parameter key is not equal to "ManagedOrganizationalUnit." The parameter value in old_ct_parameters is not equal to the parameter value in new_ct_parameters. If these conditions are satisfied for all parameters, the function returns True, indicating that the modification request is valid.

[anthony-esper-allspring]

I believe I have the root cause:

See code below:

def update_existing_account( session: Session, ct_management_session: Session, request: Dict[str, Any] ) -> None:

...

provisioning_parameters: List[UpdateProvisioningParameterTypeDef] = []
for k, v in request["control_tower_parameters"].items():
    provisioning_parameters.append({"Key": k, "Value": v})

control_tower_email_parameter = request["control_tower_parameters"]["AccountEmail"]
target_product: Optional[ProvisionedProductAttributeTypeDef] = None
for batch in get_healthy_ct_product_batch(
    ct_management_session=ct_management_session
):
    for product in batch:
        product_outputs_response = client.get_provisioned_product_outputs(
            ProvisionedProductId=product["Id"],
            OutputKeys=["AccountEmail"],
        )
        provisioned_product_email = product_outputs_response["Outputs"][0][
            "OutputValue"
        ]

        if utils.emails_are_equal(
            provisioned_product_email, control_tower_email_parameter
        ):
            target_product = product
            break

if target_product is None:
    raise Exception(
        f"No healthy provisioned product found for {control_tower_email_parameter}"
    )

# Rest of the function implementation...

}

In this function - they are retrieving the email from the SC ProvisionedProductId outputs. If you go and look at the ProvisionedProductID outputs (aws servicecatalog describe-record --id rec-xxxx), you will see that the old email address is still there. I believe this is what is causing the CT Request is not valid.

[anthony-esper-allspring]

Alright Bottom Line: We missed step 4 here: https://docs.aws.amazon.com/controltower/latest/userguide/change-account-email.html

Here is my current fix:

Remove the entries for the account inside of the account-request dynamo tables.

Need to fix the email inside of the provisioned product inside Service Catalog:

Options: 1) Update the email by updating the provisioned product for the account

or

2) Re-Enroll that account into service catalog

Rerun the import process.

I have selected option 2, once this is complete, I will let you know if the import now works.

[anthony-esper-allspring]

I have good news that my work around was successful. After the provisioned product outputs matched what was entered inside of the account-request, all was successful. Account imported, can see it inside of the metadata db and see a customization pipeline.

[anthony-esper-allspring]

@daveshepherd Check the outputs of the provisioned product for the account you had an issue with. The instructions for changing the account with CT enrolled are confusing at best. AFT will look directly at those outputs to validate the CT request. The outputs must match what is inside the account-request module.

[snebhu3]

@daveshepherd thank you for reaching out. The mismatch between SC provisioned product and current state could be causing these failures to import the account into AFT. You could align the SC provisioned product with current changes and then re-try. You might have to clean up the respective account entry from the aft-request table before re-trying. If you need further help, you may reach out to AWS premium support.

[stumins]

@anthony-esper-allspring Thank for your work isolating the specific nature of the error! There doesn't appear to be a bug with AFT here, so I'm going to go ahead and resolve this issue.

[daveshepherd]

Sorry, I've only just got back to this. Thanks @snebhu3 and @anthony-esper-allspring - I came to the same conclusion that the best way to progress was to re-enroll the account, this creates a new service catalog entry with the new/correct email address and then you can just add it to aft-account-requests with the new email and it gets imported into AFT correctly.