aft-request pipeline doesn't create pipeline for new accounts

[devalibvr]

Terraform Version & Prov:

AFT Version: 1.8.0

Terraform Version & Provider Versions

terraform version

1.3.4

terraform providers

4.28

Bug Description When we need to create a new account, our standard procedure involves merging the "aft-account-customization" first and then "aft-account-request" . This approach consistently created a new pipeline for creating the account. However, I am currently encountering an issue where the pipeline fails to generate after the successful completion of the "aft-account-request" pipeline. It's not just the pipeline that's affected; none of the state machines are executing. I attempted to resolve the problem by removing and re-adding the corresponding code segment in the "aft-request" section, hoping that the pipeline would trigger automatically. Unfortunately, no success. Could you assist me with this please?

Related Logs I can see terraform apply in ct-aft-account-request pipeline added two new account, but nothing happened after that. no state machines execution, no customization pipeline executed

Plan: 2 to add, 0 to change, 0 to destroy.

Additional context My guess is this happened after I upgraded Landing zone from v3.0 to v3.2, not sure though

[snebhu3]

@devalibvr thank you for reaching out. I suspect the account request could be failing somewhere during the process, resulting in pipeline not being created.

I would recommend:

• subscribing to the AFT failure SNS topic aft-failure-notifications and retrying to potentially help identify the cause
• upgrading AFT to latest version 1.10.3 to have all the latest bug fixes and features

If you are still facing the issue, please reach out to AWS premium support for help on troubleshooting.

[zeeamd]

I'm facing a similar issue. AFT version 1.6.7 Terraform 0.15.5

However, this issue is intermittent. I sent in multiple account request invoking the aft-account-request module. It created the pipeline but missed out on a few. The account entries exist in the aft-account-request & aft-account-request-metadata dynamo db. However, now when I re-trigger the pipeline it does not create those missing pipelines. Unfortunately, no errors in the lambda logs as well

[devalibvr]

@snebhu3 Thanks for suggestions. I've found the issue in our case. we've got an issue with AFT whereby we deleted some resources on the advice of one AWS premium support which then broke some bits of AFT .. so now another engineer is advising to remove/install AFT.

Before we undertake this drastic approach I wanted to know where the state file is for the original install of AFT i.e. the creation of state machines, lambdas etc? is that only executed locally first time and was our responsibility to store it somewhere or this is something related to AWS and stored in an account? related to what AWS engineer recommended that seems very risky to me since we have several accounts in production, and we need to understand what the consequences will be of decommissioning and implementing a new AFT. I wonder if it is possible to add a step in Terraform configuration that lets us only deploy those resources that we need. Apart from that, when we first setup AFT there was a section called "tf_backend_secondary_region", which is eu-west-3 for us. Is there a way to find out which resources we deployed for the first time and which have been deleted from AFT resources, so that we only deploy the new ones? in addition, do we need to delete all customization pipelines that create resources for each account? In that case, what will happen after recreation?

[devalibvr]

@zeeamd Have you verified whether the aft-account-provisioning-framework state machine has been executed following the successful completion of ct-aft-account-request? And could you describe the logs of the Terraform apply stage? Additionally, how many accounts did you add to ct-aft-account-request simultaneously? how many of them missed?

[zeeamd]

@zeeamd Have you verified whether the aft-account-provisioning-framework state machine has been executed following the successful completion of ct-aft-account-request? And could you describe the logs of the Terraform apply stage? Additionally, how many accounts did you add to ct-aft-account-request simultaneously? how many of them missed?

It seems like aft-account-provisioning-framework was invoked only once when multiple account requests were pushed. Somehow, it created only 1 out of the 6 account request pipelines. The invocations logs show only 1 account number on the vended account variable list. Assume need to send in only 1 request at a time. Will need to test this scenario again for confirmation. Before that need to figure out how to create those other missing pipelines.

[devalibvr]

@zeeamd Have you verified whether the aft-account-provisioning-framework state machine has been executed following the successful completion of ct-aft-account-request? And could you describe the logs of the Terraform apply stage? Additionally, how many accounts did you add to ct-aft-account-request simultaneously? how many of them missed?

It seems like aft-account-provisioning-framework was invoked only once when multiple account requests were pushed. Somehow, it created only 1 out of the 6 account request pipelines. The invocations logs show only 1 account number on the vended account variable list. Assume need to send in only 1 request at a time. Will need to test this scenario again for confirmation. Before that need to figure out how to create those other missing pipelines.

@zeeamd I have seen that scenario several times and found AFT can invoke 3 accounts at the same time. don't know about others, but in our case the AFT cannot create more than 3 pipelines at the same time. My suggestion is to edit an element in the custom_fields or change_requested_by = "" section that has no impact and then merge it to the main. This action will once again trigger the aft-account-provisioning-framework and generate your pipelines.