The AWS Service Catalog Terraform Reference Engine (TRE) provides an example for you to configure and install a Terraform engine in your AWS Service Catalog administrator account. With the engine installed into your account, you can use Service Catalog as a single tool to organize, govern, and distribute your Terraform configurations within AWS. For more information about Terraform and AWS Service Catalog, see Getting started with Terraform.
TERRAFORM_OPEN_SOURCE
to EXTERNAL
product typesAWS Service Catalog is introducing important changes for supporting Terraform Open Source starting October 20, 2023. Due to changes in the business licensing for Terraform, any references to 'Terraform Open Source' will be changed. Currently, Service Catalog references ‘open-source’ language in various artifacts such as our APIs, console, documentation, and a publicly available reference engine (TRE) that can be accessed through GitHub. For more information on what changes you should make, please refer to the IMPORTANT_UPDATES.md. You can also find similar information in our public documentation.
The installation can be done from any Linux or Mac machine.
The project includes a set of scripts to automatically install a new environment or update an existing environment. The script does the following:
The automated installation requires that the following are available on your local machine. If they are not available, please follow the instructions below for manual installation.
The instance replacement process can take a long time if you have long-running provisioning operations in flight.
The automated installation assumes you are using the default profile to store your AWS Credentials. Follow https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html to set up credentials.
You can install the engine or update an existing engine by doing the following:
If you prefer to install the engine manually, perform the steps in this section. These steps do not need to be performed if you used the automated script in the previous section.
AWS_ACCOUNT_ID=<YOUR AWS ACCOUNT ID>
AWS_REGION=<YOUR REGION OF CHOICE>
The manual instructions assume you are using the default profile to store your AWS Credentials and region. Follow https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html to set up credentials and region.
lambda-functions/terraform_open_source_parameter_parser
go mod init terraform_open_source_parameter_parser
to initialize the terraform_open_source_parameter_parser Go modulego env -w GOPROXY=direct
go mod tidy
to generate the required go.mod
and go.sum
filesgo test
within the test containing directory to execute all unit tests in Govenv
by running the command python3 -m venv venv
. venv/bin/activate
Note: Once done running all python commands call deactivate
to stop using the virtual environment.
pip3 install -r lambda-functions/state_machine_lambdas/requirements.txt -t lambda-functions/state_machine_lambdas --upgrade
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 sam build
python3 -m unittest
within the test containing directory to execute all unit tests in pythonsam local invoke <Logical ID of the function>
Use the Cloudformation console or CLI to create a stack using the Bootstrap.yaml template:
```
AWS242TerraformReferenceEngine (package-root)
└── cfn-templates
└── Bootstrap.yaml
```
To use the CloudFormation CLI, run this example CLI command -
aws cloudformation create-stack --stack-name Bootstrap-TRE --template-body file://cfn-templates/Bootstrap.yaml --capabilities CAPABILITY_NAMED_IAM
This will create a bucket named terraform-engine-bootstrap-
In this step we build the Python scripts that will run on EC2 instances to manage Terraform CLI commands. Then we upload them to the bootstrap bucket created in the previous step.
wrapper-scripts
directory: cd wrapper-scripts
pip install wheel
python3 setup.py bdist_wheel
aws s3 sync dist s3://terraform-engine-bootstrap-$AWS_ACCOUNT_ID-$AWS_REGION/dist
sam deploy --s3-bucket terraform-engine-bootstrap-$AWS_ACCOUNT_ID-$AWS_REGION --stack-name SAM-TRE --capabilities CAPABILITY_NAMED_IAM
template.yaml
.
--parameter-overrides VpcCidr="<Your_VpcCidr>" PublicSubnet1CIDR="<Your_PublicSubnet1CIDR>" PrivateSubnet1CIDR="<Your_PrivateSubnet1CIDR>"
--parameter-overrides EC2InstanceType="<Your_EC2InstanceType>"
If you are updating an existing environment, you will need to replace the EC2 instances so the new instances will pick up the latest code.
To do this manually, you can run the instance replacement script.
cd bin/bash
pip3 install boto3
export AWS_REGION=<your region>
python3 replace-ec2-instances.py
Note: The instance replacement process can take a long time if you have long-running provisioning operations in flight.
If you see this error during an update of the SAM-TRE stack:
resource sg-xxxxxxxx has a dependent object (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation;
Solution: Terminate your EC2 instances named TerraformExecutionInstance. Then rerun the failed command.
TerraformEngineProvisioningHandlerLambda
and TerraformEngineTerminateHandlerLambda
Lambda functions. This will allow messages to be sent to the queues by Service Catalog but prevent the EC2 instance(s) from starting new workflows that could be affected by shutting down the EC2 instances in the following steps.ManageProvisionedProductStateMachine
and TerminateProvisionedProductStateMachine
state machines. If there are ongoing executions, wait until they are complete before proceeding to the next step. Similar to the last step, this will help ensure no ongoing executions are impacted by shutting down the EC2 instances in the following steps.TerraformEngineProvisioningHandlerLambda
and TerraformEngineTerminateHandlerLambda
Lambda functions. This step will restart processing of any messages that have been waiting in the queue during the update procedure.In addition to the steps included in this readme, more information about creating and provisioning a service catalog product can be found here: https://docs.aws.amazon.com/servicecatalog/latest/adminguide/getstarted-Terraform.html
Every product of type TERRAFORM_OPEN_SOURCE or EXTERNAL must have a launch constraint that indicates the IAM role to be used for provisioning the product's resources. This role is known as the "launch role."
An example launch role is here: cfn-templates/TerraformProvisioningAccount.yaml
The Terraform Reference Engine has the following requirements for the launch role.
In addition, Service Catalog has requirements for each launch role used for products of type TERRAFORM_OPEN_SOURCE or EXTERNAL. https://docs.aws.amazon.com/servicecatalog/latest/adminguide/getstarted-launchrole-Terraform.html
Now the Service Catalog end user can call these provisioning operations on the product:
The Terraform Reference Engine will perform these operations in the account where the product was created. If the launch role is in a different account, the resources will be provisioned in that account.
The Service Catalog Terraform Reference Engine is a project that allows product cataloging and provisioning using Terraform as the choice for infrastructure as code.
Service Catalog supports a product type called EXTERNAL. Customers use this product type to catalog and provision products where the implementation of the product is built by the customer. This product implementation is known as an engine.
In this reference architecture, we have built an External product engine for Terraform. The provisioning artifacts will be Terraform configuration files written in Hashicorp Config Language (HCL). The creation and management of resources will be done using Terraform running on EC2 instances.
The Terraform reference engine includes a Lambda function to parse variables in the provisioning artifact and return them to Service Catalog. The function is invoked when the end user calls the Service Catalog DescribeProvisioningParameters API.
The Service Catalog Terraform Parameter Parser is a Lambda function written in Go. It uses the Hashicorp Config Inspect library to inspect HCL files: https://github.com/hashicorp/terraform-config-inspect
Only files with .tf filename extension in the zipped .tar.gz provisioning artifact would be parsed. Files in the .tf.json format are not supported.
The Service Catalog Terraform Parameter Parser assumes the provided launch role to download the artifact if a launch role is provided in the payload. If no launch role is provided, parameter parser will use the default lambda execution role ServiceCatalogTerraformOSParameterParserRole credentials to download the artifact.
If provided, the launch role arn must be a valid IAM arn that has access to the artifact and is assumable by the parser lambda.
The Service Catalog Terraform Parameter Parser parses files with override.tf suffix as override files
Override files are loaded after normal configuration files with the merging behavior for parameters similar to the one described in Terraform documentation https://developer.hashicorp.com/terraform/language/files/override
Behavior when multiple override files define the same top-level variable block is undefined
Please also refer to the Override Files section under Limitations in the README below to understand the risks of using override files in Terraform Reference Engine
The Service Catalog Terraform Parameter Parser throws two types of exceptions: ParserInvalidParameterException and ParserAccessDeniedException
ParserInvalidParameterException is thrown when the provided input is invalid
ParserAccessDeniedException is thrown when the provided launch role cannot be assumed by the parser, or the artifact cannot be accessed with the launch role
The Terraform reference engine contains workflows that implement Service Catalog provisioning functions.
Each provisioning workflow consists of the following components:
AWS Service Catalog publishes a message to an SQS queue for each provisioning operation.
Operation | Queue Name |
---|---|
Provision | ServiceCatalogTerraformOSProvisionOperationQueue |
Update | ServiceCatalogTerraformOSUpdateOperationQueue |
Terminate | ServiceCatalogTerraformOSTerminateOperationQueue |
The Terraform reference engine contains Lambda message handler functions that consume messages from the inbound queues. The job of each function is to use the message contents as input to start the corresponding workflow's state machine.
Note that there is one handler for both provision and update messages. This is because provision and update operations use the same provision/update workflow. Terminate messages have a separate handler and a separate workflow.
Each workflow has a Step Functions state machine responsible for overall logic of the workflow. Although they have some differences, the provision/update state machine and terminate state machine follow similar steps.
The Run Command execution starts a script on one of the EC2 instances to do the work of Terraform apply or destroy, depending on the workflow. In the Terraform Reference Engine, a Python package named terraform_runner handles the CLI commands to complete Terraform apply or destroy.
For the provision/update workflow, the terraform_runner package performs these steps.
For the terminate workflow, the terraform_runner package performs these steps.
Unit tests are included for each Lambda function as well as for the terraform_runner Python package.
All provisioning artifacts must be in tar.gz format and have a filename extension of .tar.gz. Inside the tar.gz file, the Terraform config files must be in .tf format. The .tf.json format is not supported.
When creating the tar.gz file, be sure that the root-module files are at the root directory of the archive. Some archiving tools default to putting the files into a top-level directory. If that happens, the engine cannot determine the root module from local modules in subdirectories.
The Terraform Reference Engine adds some override files to each configuration before running Terraform CLI commands. Therefore, if your configuration conflicts with any of the below overrides, the behavior of the CLI commands on that configuration will be undefined. We strongly recommend that you avoid including override files in your provisioning artifact.
The engine adds a file named backend_override.tf.json. This sets the Terraform state bucket S3 to the state bucket created from the Sam template and sets the S3 key to a name that uniquely identifies the provisioned product.
Example:
{
"terraform": {
"backend": {
"s3": {
"bucket": "sc-terraform-engine-state-<your account ID>",
"key": "pp-1234",
"region": "us-east-1"
}
}
}
}
The engine does not include DynamoDB state locking in the backend configuration. This is not needed for two reasons.
The engine adds a file named variable_override.tf.json. This file sets variable defaults based on the parameters provided by the end user to Service Catalog when performing a provision or update operation.
Example:
{
"variable": {
"key1": {
"default": "parameter-value1"
},
"key2": {
"default": "parameter-value2"
}
}
}
The engine adds a file named provider_override.tf.json. The engine sets several overrides in this file:
Example:
{
"provider": {
"aws": {
"region": "us-east-1",
"assume_role": {
"role_arn": "arn:aws:iam::111122223333:role/SCLaunchRoleTerraformExample",
"session_name": "pp-1234_MyProvisioned_product"
},
"default_tags": {
"tags": {
"SERVICE_CATALOG_TERRAFORM_INTEGRATION-DO_NOT_DELETE": "pp-1234"
}
}
}
}
}
The engine overrides the "aws" provider. Other providers are not supported.
If you do include other providers in your Terraform configuration, be sure they are not writing to the local file system outside the current working directory. The engine creates a unique working directory for each provisioned product on the EC2 instance. If the Terraform configuration writes outside this directory, those files could interfere with processes working on other provisioned products on the same instance.
The tag SERVICE_CATALOG_TERRAFORM_INTEGRATION-DO_NOT_DELETE will be applied to all resources provisioned by the engine. Do not modify this tag. It is required for Service Catalog to maintain a resource group containing the resources for the provisioned product.
Service Catalog will time out your provisioning operation if it runs too long. See the Service Catalog documentation for the maximum duration of a provisioning operation.
If you include resource timeouts in your Terraform configuration, the timeout with the shorter duration will take effect.
Due to AWS Lambda memory size constraint, large provisioning artifacts compressed using standard .tar.gz algorithm over 500 KB might fail to be parsed.
MemorySize setting for the parameter parser can be adjusted up to 10240 MB by overriding ParameterParserLambdaMemorySize parameter in the template in order to accommodate the need for parsing very large provisioning artifacts.
Parameter parser parses Terraform variable arguments including name, default, type, description and sensitive. Specifically, validation and nullable fields are not parsed.