Optional custom fields for customizations

[casimcdaniels]

Currently the only documented way to pass properties to the customization pipeline from the account request is to add them to the custom_fields object within the AFT account requests. Once this is done, an SSM parameter with the field_name is created within the target account. In account customizations, the field can be read using an aws_ssm_parameter data source and used to configure the customization. While this works fine it doesn't work so great for optional configuration parameters due to the nature in how data resources function in the AWS provider. Specifically, if the field isn't provided in the account request, the pipeline fails completely due to the missing SSM parameter. Without any alerting, this can be annoying to debug.

Workaround As a solution to this I've defined wrapper modules for each customization in my AFT account requests repo to abstract away the custom_fields into options. For example, if my customization allows for optional Jenkins integration, then I can pass a jenkins_host, which passes either an empty string or some configuration value that can be read from the customization to configure an allowed inbound security group. This works for me as a solution though I find it a restrictive workflow as any changes in the customization need to be synchronized to the wrapper module and vice versa and passing an empty string is annoying to prevent the missing SSM parameter issue.

Ideas

1. It would be nice to be able to have custom_fields or possibly another variable (tf_vars?) populate TF_VARS_* or a .tfvars in the pipeline so we can directly access and control values natively in terraform without needing to go through SSM at all. Additionally, since they are passed as TF_VARS we can define variable blocks in our customization with types in mind.

2. Allow for sourcing environment variables within our pre or post build scripts by fetching from the metadata table or SSM then exporting e.g. export TF_VARS_jenkins_host=${JENKINS_HOST}. Currently the pre/post scripts are run as ./pre_build_script.sh which isolates any environment variables to the child shell.

[rikturnbull]

We struggled with this too, using aws_ssm_parameter. We also hit SSM throttling limits populating parameter store with a bunch of unused default values. After trying out modules we ended up with this pattern, using aws_ssm_parameters_by_path, inside customizations:

data "aws_ssm_parameters_by_path" "custom_fields" {
  path = "/aft/account-request/custom-fields"
}

locals {
  defaults = {
    global_foo = "bar"
  }

  custom_fields = merge(local.defaults,
    { for i, v in data.aws_ssm_parameters_by_path.custom_fields.names : element(split("/", v), 4) => data.aws_ssm_parameters_by_path.custom_fields.values[i] }
  )
}

So global_foo defaults to bar unless it is overwritten from a value in parameter store from the account request. You can then safely reference local.custom_fields.global_foo or do some validation. This is our best workaround so far.