I took all the feedback, issues and feature requests from all our users to create this new major version. All known bugs for CommandRunner have now been fixed!
This version comes with 3 new properties InstanceType, Timeout and DisableTerminateInstancesCheck, improved error handling, logging, reliability, documentation and functionality.
To update to the new version or do a fresh install, simply run the following commands in a new directory.
git clone https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner.git
cd aws-cloudformation-resource-providers-awsutilities-commandrunner
curl -LO https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner/releases/latest/download/awsutility-cloudformation-commandrunner.zip
./scripts/register.sh --set-defaultFor more details, check the Change Log section below.
AWSUtility::CloudFormation::CommandRunner is a CloudFormation resource type created using the recently released CloudFormation Resource Providers framework.
The AWSUtility::CloudFormation::CommandRunner resource allows users to run Bash commands in any CloudFormation stack.
This allows for unlimited customization such as executing AWS CLI/API calls, running scripts in any language, querying databases, doing external REST API calls, cleanup routines, validations, dynamically referencing parameters and just about anything that can be done using the shell on an EC2 instance.
The AWSUtility::CloudFormation::CommandRunner resource runs any command provided to it before or after any resource in the Stack.
AWSUtility::CloudFormation::CommandRunner can be used to perform inside your CloudFormation stack, any API call, script, custom logic, external check, conditions, cleanup, dynamic parameter retrieval and just about anything that can be done using a command.
Any output written using the command to the reserved file /command-output.txt can be referenced anywhere in your template by using !Fn::GetAtt Command.Output like below, where Command is the logical name of the AWSUtility::CloudFormation::CommandRunnerresource.
Resources:
MyCommand:
Type: 'AWSUtility::CloudFormation::CommandRunner'
Properties:
Command: aws s3 ls | sed -n 1p | cut -d " " -f3 > /command-output.txt
Role: String #Optional
LogGroup: String #Optional
SubnetId: String #Optional
SecurityGroupId: String #Optional
KeyId: String #Optional
Timeout: String #Optional **NEW**
DisableTerminateInstancesCheck: String #Optional **NEW**
InstanceType: #Optional **NEW**
Outputs:
Output:
Description: The output of the CommandRunner.
Value: !GetAtt MyCommand.OutputNote: In the above example, sed -n 1p prints only the first line from the response returned by aws s3 ls. To get the bucket name, sed -n 1p pipes the response to cut -d " " -f3, which chooses the third element in the array created after splitting the line delimited by a space.
Only the property Command is required, while Role, LogGroup, SubnetId and SecurityGroupId are not required and have defaults.
Command is the Bash command.
Role is the IAM Role to run the command.
LogGroup is the CloudWatch Log Group to send logs from the command's execution.
SubnetId is the ID of the Subnet that the command will be executed in.
SecurityGroupId is the ID of the Security Group applied during the execution of the command.
For more information about the above properties, navigate to Properties in the Documentation.
Note that the command once executed cannot be undone. It is highly recommended to test the AWSUtility::CloudFormation::CommandRunner resource out in a test stack before adding it to your production stack.
AWSUtility::CloudFormation::CommandRunner resource to your AWS account/region, you need IAM permissions to perform the following actions.s3:CreateBucket
s3:DeleteBucket
s3:PutBucketPolicy
s3:PutObject
cloudformation:RegisterType
cloudformation:DescribeTypeRegistration
iam:createRole
logs:CreateLogGroupNote: To build the source yourself, see the Developer Build Steps section below.
Step 0: Clone this repository and download the latest release using the following commands.
git clone https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner.git
cd aws-cloudformation-resource-providers-awsutilities-commandrunner
curl -LO https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-awsutilities-commandrunner/releases/latest/download/awsutility-cloudformation-commandrunner.zipStep 1: Use the register.sh bash script to register resource from scratch and upload package to S3 bucket. Pass the optional --set-default option to set this version to be the default version for the AWSUtility::CloudFormation::CommandRunner resource.
$ ./scripts/register.sh --set-default...And that's it!
Below is an example of a successful registration using the register.sh script.
$ ./scripts/register.sh
Creating Execution Role...
Waiting for execution role stack to complete...
Waiting for execution role stack to complete...
Waiting for execution role stack to complete...
Waiting for execution role stack to complete...
Creating/Updating Execution Role complete.
Creating temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2...
Creating temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2 complete.
Configuring S3 Bucket Policy for temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2...
Configuring S3 Bucket Policy for temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2 complete.
Copying Schema Handler Package to temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2...
Copying Schema Handler Package to temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2 complete.
Creating CommandRunner Log Group called awsutility-cloudformation-commandrunner-logs2...
Creating CommandRunner Log Group complete.
Registering AWSUtility::CloudFormation::CommandRunner to AWS CloudFormation...
RegistrationToken: 0ae0622e-af3d-463b-9b2d-1d1e5fa41d14
Waiting for registration to complete...
Waiting for registration to complete...
Waiting for registration to complete...
Waiting for registration to complete...
Registering AWSUtility::CloudFormation::CommandRunner to AWS CloudFormation complete.
Cleaning up temporary S3 Bucket...
Deleting SchemaHandlerPackage from temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2...
Deleting SchemaHandlerPackage from temporary S3 Bucket 7c96b969af1c41bfb2bd10f552255ca2 complete.
Cleaning up temporary S3 Bucket complete.
AWSUtility::CloudFormation::CommandRunner is ready to use.The register.sh script performs the following operations to register the AWSUtility::CloudFormation::CommandRunner resource.
AWSUtility::CloudFormation::CommandRunner resource to give it permissions to perform the following actions.cloudformation:DeleteStack
cloudformation:CreateStack
cloudformation:DescribeStacks
logs:CreateLogStream
logs:DescribeLogGroups
logs:PutLogEvents
cloudwatch:PutMetricData
ssm:GetParameter
ssm:PutParameter
ssm:DeleteParameter
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeSecurityGroups
ec2:CreateSecurityGroup
ec2:RevokeSecurityGroupEgress
ec2:RevokeSecurityGroupIngress
ec2:CreateTags
ec2:AuthorizeSecurityGroupIngress
ec2:AuthorizeSecurityGroupEgress
ec2:RunInstances
ec2:DescribeInstances
ec2:TerminateInstances
ec2:DeleteSecurityGroup
iam:PassRole
iam:GetInstanceProfile
iam:SimulatePrincipalPolicy
#Only required if using the KeyId property, i.e custom KMS Key for the SSM SecureString
kms:Encrypt
kms:Decrypt
sts:GetCallerIdentityaws s3 mb AWS CLI command to create an S3 bucket with the name specified.aws s3api put-bucket-policy AWS CLI command to put the following bucket policy on the new bucket, where <BUCKET_NAME> is the specified bucket name.{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>/*",
"arn:aws:s3:::<BUCKET_NAME"
],
"Principal": {
"Service": "cloudformation.amazonaws.com"
}
}
]
}aws s3 cp AWS CLI command to copy the SchemaHandlerPackage awsutility-cloudformation-commandrunner.zip into the S3 bucket.aws logs create-log-group AWS CLI command to create the Log Group or skip it if it already exists.aws cloudformation register-type AWS CLI command to register the AWSUtility::CloudFormation::CommandRunner resource to CloudFormation.Note that to run the register.sh script the IAM Role/User configured in the AWS CLI should have the following IAM permissions.
s3:CreateBucket
s3:DeleteBucket
s3:PutBucketPolicy
s3:PutObject
cloudformation:RegisterType
cloudformation:DescribeTypeRegistration
iam:CreateRoleThe following sample policy can be attached to the IAM User/Role if they do not have the necessary permissions.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudformation:RegisterType",
"cloudformation:DescribeTypeRegistration"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:PutBucketPolicy",
"s3:PutObject"
],
"Resource": "*"
}
]
}{
"Resources": {
"CommandRunner": {
"Type": "AWSUtility::CloudFormation::CommandRunner",
"Properties": {
"Command": "String",
"Role": "String",
"LogGroup": "String",
"SubnetId": "String",
"SecurityGroupId": "String",
"KeyId": "String",
"Timeout": "String",
"DisableTerminateInstancesCheck": "String",
"InstanceType": "String"
}
}
}
}Resources:
CommandRunner:
Type: 'AWSUtility::CloudFormation::CommandRunner'
Properties:
Command: String
Role: String #Optional
LogGroup: String #Optional
SubnetId: String #Optional
SecurityGroupId: String #Optional
KeyId: String #Optional
Timeout: String #Optional **NEW**
DisableTerminateInstancesCheck: String #Optional **NEW**
InstanceType: #Optional **NEW**
Outputs:
Output:
Description: The output of the CommandRunner.
Value: !GetAtt Command.Output
The bash command that you would like to run.
For AWS CLI commands, please specify the region using the --region option.
Every command needs to output the desired value into the reserved file "/command-output.txt" like the following example. The value written to the file must be a non-empty single word value without quotation marks like vpc-0a12ab123abc9876 as they are intended to be used inside the CloudFormation template using Fn::GetAtt.
aws ec2 describe-vpcs --query Vpcs[0].VpcId --output text > /command-output.txt
The command is run on the latest Amazon Linux 2 AMI in your region.
Required: Yes
Type: String
Update requires: Replacement
The IAM Instance Profile to be used to run the Command. The Role in the Instance Profile will need all the permissions required to run the above Command.
The Role should have permissions to perform the actions below to write logs to CloudWatch from the command's execution.
"logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"If the Role does not have the above logging permissions, the command will still work but no logs will be written.
The Role in the Instance Profile should specify ec2.amazonaws.com as a Trusted Entity.
An Instance Profile is created automatically when a Role is created using the Console for an EC2 instance.
Required: No
Type: String
Update requires: Replacement
The CloudWatch Log Group to stream the logs from the specified command.
If one is not provided the default cloudformation-commandrunner-log-group one will be used.
If the specified log group does not exist, a new one will be created.
To log a trace of your commands and their arguments after they are expanded and before they are executed, run set -xe in the Command property before your actual command.
Required: No
Type: String
Update requires: Replacement
The Id of the Subnet to execute the command in. Note that the SubnetId specified should have access to the internet to be able to communicate back to CloudFormation. Ensure that the Route Table associated with the Subnet has a route to the internet via either an Internet Gateway (IGW) or a NAT Gateway (NGW).
If the SubnetID is not specified, it will create the resource in a subnet in the default VPC of the region.
Required: No
Type: String
Update requires: Replacement
The Id of the Security Group to attach to the instance the command is run in. If using SecurityGroup, the SubnetId property is required.
If the SecurityGroupId is not specified, the command will be run with a security group with open Egress rules and no Ingress rules.
Required: No
Type: String
Update requires: Replacement
Id of the KMS key to use when encrypting the output stored in SSM Parameter Store. If not specified, the account's default KMS key is used.
Required: No
Type: String
Update requires: Replacement
By default, the timeout is 600 seconds. To increase the timeout specify a higher Timeout value in seconds. The maximum timeout value is 43200 seconds i.e 12 hours.
Required: No
Type: String
Update requires: Replacement
By default, CommandRunner checks to see if the execution role can perform a TerminateInstances API call. Set this property to true if you want to skip the check. Note that this means that the CommandRunner instance may not be terminated and will have to be terminated manually.
Required: No
Type: String
Update requires: Replacement
By default, the instance type used is t2.medium. However you can use this property to specify any supported instance type.
Required: No
Type: String
Update requires: Replacement
Users can reference the output of the command written to /command-output.txt using Fn::GetAtt like in the following syntax.
Outputs:
Output:
Description: The output of the command.
Value: !GetAtt Command.OutputTo run the command after a resource with logical name Resource, specify DependsOn: Resource in the AWSUtility::CloudFormation::CommandRunner resource's definition.
Resources:
Command:
DependsOn: Resource
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command: echo success > /command-output.txt
LogGroup: my-cloudwatch-log-group
Role: MyEC2InstanceProfile
Resource:
Type: AWS::EC2::Instance
Properties:
Image: ami-abcd1234To run the command before a resource, put a DependsOn with the logical name of the AWSUtility::CloudFormation::CommandRunner resource in that resource's definition.
Resources:
Command:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command: echo success > /command-output.txt
LogGroup: my-cloudwatch-log-group
Role: MyEC2InstanceProfile
Resource:
DependsOn: Command
Type: AWS::EC2::Instance
Properties:
Image: ami-abcd1234You can write a script in any programming language and upload it to S3. Use the aws s3 cp command to copy the script from S3 followed by && and the command to run the script like the following example.
Resources:
Command:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command: 'aws s3 cp s3://cfn-cli-project/S3BucketCheck.py . && python S3BucketCheck.py my-bucket third-name-option-a'
Role: MyEC2InstanceProfile
LogGroup: my-cloudwatch-log-group
Outputs:
Output:
Description: The output of the command.
Value: !GetAtt Command.OutputResources:
Command:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command:
Fn::Sub: |
yum install jq -y
aws ssm get-parameter --name RepositoryName --region us-east-1 > response.json
jq -r .Parameter.Value response.json > /command-output.txt'
Role: MyEC2InstanceProfile
LogGroup: my-cloudwatch-log-group
Outputs:
Output:
Description: The output of the command.
Value: !GetAtt Command.OutputResources:
Command:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command:
Fn::Sub: |
aws ssm get-parameter --name RepositoryName --region us-east-1 --query Parameter.Value --output text > /command-output.txt
Role: MyEC2InstanceProfile
LogGroup: my-cloudwatch-log-group
Outputs:
Output:
Description: The output of the command.
Value: !GetAtt Command.OutputResources:
CommandRunner:
Type: AWSUtility::CloudFormation::CommandRunner
Properties:
Command:
Fn::Sub: |
echo "my log"
echo '{"key":"value"}' > mydata.json
...
<ANY OTHER COMMANDS>
...
echo success > /command-output.txt
LogGroup: my-cloudwatch-log-group
Role: MyEC2InstanceProfileThe AWSUtility::CloudFormation::CommandRunner resource lets you perform any API call, script, custom logic, external check, conditions, cleanup, dynamic parameter retrieval and anything else that can be done using a command.
Get parameters dynamically during the Stack's execution instead of passing in Parameters during stack creation.
Currently, there is no AWS::ECS resource that allows you to configure the Account Settings. However, you can do this using the AWSUtility::CloudFormation::CommandRunner resource by running the aws ecs put-account-setting CLI commmand.
Currently, there is no way to create an image (AMI) using a running EC2 instance, but it can be done using the AWSUtility::CloudFormation::CommandRunner resource by using the aws ec2 create-image CLI command.
Add a lag between resources using CommandRunner. Specify a sleep command in the Command property.
Lambda does not natively support using Bash.
Even if Bash is added using custom Lambda Layers it will still not allow installing new packages or running other programming / scripting languages.
Lambda is also more expensive than running a small EC2 instance for approximately 2 minutes.
Developing a Custom Resource or Macro requires writing several lines of code and troubleshooting which is considered to be development effort.
The AWSUtility::CloudFormation::CommandRunner provides a quick fix to solve problems with a single line of code and does not require any development effort.
Using the AWSUtility::CloudFormation::CommandRunner, users can quickly unblock themselves without relying on CloudFormation to resolve the issue to unblock them.
Please execute the included build script by running ./scripts/build.sh to build and register the resource to AWS CloudFormation for your account. The script waits while CloudFormation registers the resource so it typically takes about 5-10 minutes.
Note that the script assumes that you have AWS CLI configured and the necessary permissions to register a resource provider to CloudFormation, jq, mvn and the prerequisites mentioned here. https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/what-is-cloudformation-cli.html
The build script will do the following:
Runs cfn generate to generate the rpdk files for the Resource Provider.
mvn package packages the Java code up and cfn submit registers the built Resource to AWS CloudFormation in your AWS account.
From the output of the cfn submit command, it gets the version of the build and updates the default version to be used in CloudFormation.
Once the script finishes, the AWSUtility::CloudFormation::CommandRunner resource will be ready to use.
You can find an example of how to use the resource in the file usage-template.yaml.
As of March 2024, the recommended versions and dependencies for the build are as follows.
$ cfn --version
cfn 0.2.35
$ mvn -version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/homebrew/Cellar/maven/3.9.6/libexec
Java version: 19.0.2, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-19.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.1.1", arch: "aarch64", family: "mac"
$ java -version
java 19.0.2 2023-01-17
Java(TM) SE Runtime Environment (build 19.0.2+7-44)
Java HotSpot(TM) 64-Bit Server VM (build 19.0.2+7-44, mixed mode, sharing)
$ ./scripts/build.shjava8 to java8.al2.register.sh.register.sh./command-output.txt
/command-output.txt to CloudWatch logs under cloud-init-output.log.register.sh
register.sh, along with handling the case where it already exists.SubnetId, SecurityGroupId
Timeout property.
DisableTerminateInstancesCheck property.
ec2:TerminateInstances action, but they still want to create CommandRunner instances. Setting this property to true allows them to create CommandRunner instances even without the ec2:TerminateInstances action.InstanceType property./command-output.txt.cloudwatch: to logs:, fixing the permissions issue when writing logs.SecureString by default i.e Encrypted at rest using the Default KMS key of the account.KMSKeyId allowing users to specify their own customer-managed KMS Key to encrypt the SSM SecureString Parameter."logs:CreateLogStream",
"logs:CreateLogGroup",
"logs:PutLogEvents"register.shand user build stepscfn test all work with the following results.collected 12 items / 5 deselected / 7 selected
handler_create.py::contract_create_delete PASSED [ 14%]
handler_create.py::contract_create_duplicate PASSED [ 28%]
handler_create.py::contract_create_read_success PASSED [ 42%]
handler_delete.py::contract_delete_read PASSED [ 57%]
handler_delete.py::contract_delete_delete PASSED [ 71%]
handler_delete.py::contract_delete_create SKIPPED [ 85%]
handler_misc.py::contract_check_asserts_work PASSED [100%]build.sh, it now does not use S3. The BaseTemplate.json is stored as a resource in the .jar file.Removed all the extra build steps, now it takes only running the build.sh script after you’ve cloned the repo.
CreateHandler.java needed to be replaced. They are now dynamically inferred and replaced in CreateHandler.java using the build.sh script.SubnetId and the SecurityGroupId or both. If neither is provided it’ll use a subnet in the default VPC and create a SecurityGroup automatically.Instead of using {{resolve}}, Fn::GetAtt now works, you can do !GetAtt Command.Output
Updated the presentation to reflect the above changes.
Updated the documentation
build.sh doesIf a security group is not provided, the one that is automatically created has no inbound rules and only allows outbound communication.
Previously I had provided wildcard permissions to the resource. Now, only the below permissions are used and there are no wildcards.
cloudformation:DeleteStack
cloudformation:CreateStack
cloudformation:DescribeStacks
logs:CreateLogStream
logs:DescribeLogGroups
ssm:GetParameter
ssm:PutParameter
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:DescribeSecurityGroups
ec2:CreateSecurityGroup
ec2:RevokeSecurityGroupEgress
ec2:RevokeSecurityGroupIngress
ec2:CreateTags
ec2:AuthorizeSecurityGroupIngress
ec2:AuthorizeSecurityGroupEgress
ec2:RunInstances
ec2:DescribeInstances
ec2:TerminateInstances
ec2:DeleteSecurityGroup
iam:PassRole
#Only required if using the KeyId property, i.e custom KMS Key for the SSM SecureString
kms:Encrypt
kms:Decrypt