Customization pipeline doesn't exist

[mathewmoon]

When pushing to aft-account-customizations no pipeline is triggered. Looking in Codecommit I see a pipeline for account requests and account provisioning customizations and no others. When trying to run the aft-invoke-customizations state machine manually to apply customizations I get the following error from the aft-customizations-execute-pipeline Lambda:

    "time_stamp": "2022-02-01 21:53:15,558",
    "log_level": "ERROR",
    "log_message": {
        "FILE": "aft_customizations_execute_pipeline.py",
        "METHOD": "lambda_handler",
        "EXCEPTION": "Pipelines for account id ************ was not found"
    }
}
Traceback (most recent call last):
  File "/var/task/aft_customizations_execute_pipeline.py", line 101, in lambda_handler
    execute_pipeline(session, str(p))
  File "/var/task/aft_customizations_execute_pipeline.py", line 70, in execute_pipeline
    name = get_pipeline_for_account(session, account)
  File "/var/task/aft_customizations_execute_pipeline.py", line 40, in get_pipeline_for_account
    raise Exception("Pipelines for account id " + current_account + " was not found")
Exception: Pipelines for account id ************ was not found

I tried updating the AFT module to 1.2.1 but still no luck. The account ID in the exception is the AFT Management account.

[mathewmoon]

Looking at Event Bridge, there is a rule for the aft-account-request and aft-account-provisioning-customizations repositories but none for the others.

[jeremyciak]

Have you provisioned any accounts yet? From what I can tell, a new pipeline is created per account that should get triggered to run when a change is detected from the aft-account-customizations or aft-global-customizations repositories. My problem is that I deployed an account without having any customizations defined, and I am trying to add customizations now and I can't get anything to automatically trigger from changes made in a proper GitOps fashion.

[mathewmoon]

Same issue here.

[mathewmoon]

@jeremyciak do you know what Lambda creates the pipelines? And why would it need a pipeline for the AFT account? I specified a different account as the target in the state machine.

[jeremyciak]

@mathewmoon it appears that the magic you seek lies here: https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/sources/aft-customizations-common/templates/customizations_pipeline/codepipeline.tf

However, this Terraform is distinctly lacking some components you will find here, which generates the other pipelines: https://github.com/aws-ia/terraform-aws-control_tower_account_factory/blob/main/modules/aft-code-repositories/codepipeline.tf

If we could get the same kind of aws_cloudwatch_event_rule and aws_cloudwatch_event_target resources defined so that the pipeline runs automatically in a proper GitOps fashion I think this would address the problem at hand.

[jeremyciak]

And the aft-account-provisioning-framework step function is what calls the aft-create-pipeline build project which runs the Terraform code I linked above.

[mathewmoon]

Thanks. I have to dig through the code to see where this is called. I feel the same about the events. I'm guessing there is a reason it wasn't done that way???

[jeremyciak]

I would speculate it's an oversight/bug and not intentional, but I could be wrong. I would submit a PR with my proposed changes but I've done that once previously and was told they're not accepting PR's.

[txsutton]

I am getting the same issue, when running the step function aft-invoke-customizations, this just runs through and completes quickly as it cannot find anything that matches my account id (as it is not in the aft-request-metadata dynamodb table).

I believe the reason it is not in the DB is because the aft-account-provisioning-framework step function doesn't create the pipeline as the input states

"account_provisioning": { "run_create_pipeline": "false" }

If I change it true and re-run the step-function than the pipeline is created and I can use the customisations

[mathewmoon]

@txsutton This seems to have fixed it. I have a pipeline running for an account now. Have you seen any side effects of doing this manually?

[txsutton]

I have only run it once for one account, so can't comment if it has any side affects yet but it hopefully work until they fix the issue

[jeremyciak]

What is odd is that the customization pipeline was in fact created automatically for me but would not run when I pushed code changes due to the issues I outlined above how it is missing CloudWatch event triggers.

[txsutton]

@jeremyciak - from my understanding, if you want to change the customization after the baseline you would have to execute it through the step function (https://docs.aws.amazon.com/controltower/latest/userguide/aft-account-customization-options.html ) or if it just a single account than just run the pipeline manually.

[jeremyciak]

@txsutton Thank you for pointing me to the documentation I was unable (or unwilling?) to find previously. So this is indeed "by design". I'm still a bit unsure why the implementation wouldn't simply have the same trigger mechanism so that customization changes followed the GitOps model as well.

[ojacques]

The re:Invent video goes into more explanations.

• Global customizations not executed automatically, by design: https://youtu.be/8Ot5wn7kxI0?t=1680
• Account customizations not executed automatically by design: https://youtu.be/8Ot5wn7kxI0?t=2140

[snebhu3]

Thank you for reaching out. Please find some clarifications to the questions / concerns called out above:

• According to current design (documentation), the account customization pipeline is triggered via a step function which you can execute. The reason we do not have the customization pipeline auto-triggered is to give you the control in case you want to do rolling deployments.

• The error message "Pipelines for account id ************ was not found" is incorrectly indicating the AFT management account instead of the target vended account, and we have a backlog to fix this error messaging.

[txsutton]

My concern was more why the pipeline didn't exist as I expect that on running the ct-aft-account-request pipeline it would create all the customizations-pipelines ready to go.

I followed my process of fixing the ones that had failed in the aft-account-provisioning step function as I mentioned above and found a dynamodb trigger for aft-request had been disabled due to the pipeline and so re-enabled that and it is now looking better, after changing the change_management_parameters on the aft-account-request repo it seems to have kicked them back into life and started creating pipelines as I would expect.

[snebhu3]

The account customization pipeline gets created after the account has been added to Service Catalog and is the last step of the aft-account-provisioning-framework Step Function. If an earlier part of the Step Function fails, it would prevent the customization pipeline from being created. Let me know if there's anything I misunderstood from the issue and I can dive deeper if so.

[mathewmoon]

For anyone who is still unclear the full resolution to my original issue was:

• Following the comment from @txsutton I ran the aft-account-provisioning-framework step function with run_create_pipeline set to true will create a pipeline for an account that does not have one.
• The pipeline will run global customizations, account customizations, and their respective api helpers
• To rerun account customizations (global or otherwise) for a specific account you can run the pipeline manually from CodePipeline
• You can target multiple accounts/ous at once with the aft-invoke-customizations step function. For each account targeted the pipeline will be run.

It's worth noting that there is a step in the documentation on AWS's site for initially configuring AFT that points out that the ct-aft-account-provisioning-customizations CodePipeline will fail when the AFT module is first deployed and should be run manually afterward, at which point it will succeed. This (at least when I deployed) was not in the docs in the repository. I'm not sure it that's the reason that even my global customizations didn't fire originally or not.

Also originally the documentation for account requests stated that if a particular flag wasn't set in the account request then you wouldn't be able to run a customization at a later time. This turned out to be wrong. The flag mentioned actually wasn't an input in the source code. This line was removed from the README but not the AWS website. I'm not sure if leaving the customizations name empty when the account is created has this effect though. Going forward I am creating a customization for ALL accounts, even if it is nothing but an empty TF file.

[snebhu3]

@mathewmoon Thank for reaching out. Glad to know that you were able to fix the issue you were facing.

Also, thanks for calling out the 'create_customization' flag requirement was removed from README but was not removed from the public document. I have created an internal backlog to fix that.

I will go ahead a close this issue for now. Please feel free to reach out in case of further questions / concerns.