[aws-ec2] cfn-init installation hooks

cprice404-aws commented 4 years ago

When I add an init section to my EC2 instance that I am booting using an Amazon Linux 2 AMI, the stack fails to deploy because CFN never receives the signal callback from the node.

It appears to me that the root cause is that adding the init section in CDK causes some UserData to be added to make a call to /opt/aws/bin/cfn-init (which is great!), but on this AMI, that script is not installed. So we probably need to do a yum install before the call to cfn-init... I can't immediately see how to do that, as calls to "addUserData" after the instance object is instantiated result in adding new things to userdata after the cfn-init command.

Reproduction Steps

const instance = new ec2.Instance(this, 'MyInstance', {
      instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.MICRO),
      machineImage: ec2.MachineImage.lookup({
        name: 'amzn2-ami-ecs-hvm-2.0.20200813-x86_64-ebs',
      }),
      vpc: vpc,
      securityGroup: securityGroup,
      init: ec2.CloudFormationInit.fromElements(
          ec2.InitCommand.shellCommand('sudo yum install tmux'),
      ),
      initOptions: {
        timeout: cdk.Duration.minutes(15)
      }
    })

What did you expect to happen?

Node to boot up and have executed the init commands.

What actually happened?

Stack deploy failed after the cfn init timeout expired.

Environment

CLI Version : 1.59.0
Framework Version: 1.59.0
Node.js Version: 14.7.0
OS : OSX for cdk env, AL2 for EC2 instance
Language (Version): TypeScript

Other

This is :bug: Bug Report

rix0rrr commented 4 years ago

According to this page the scripts should be installed on recent ("the latest") Amazon Linux versions.

How did you pick your current version? How old is it? Can you update?

cprice404-aws commented 4 years ago

I used amzn2-ami-ecs-hvm-2.0.20200813-x86_64-ebs. It was the most recent version, AFAIK, at the time that I filed this bug.

$ ls -l /opt/aws/bin
total 0
lrwxrwxrwx 1 root root 21 Aug 14 21:32 ec2-metadata -> /usr/bin/ec2-metadata

frankisans commented 3 years ago

In addition to Amazon Linux, this problem also occurs with other Linux instances that do not come pre-installed with the CloudFormation helper scripts. If we add a CloudFormation Init configuration to the instance Construct using the init property, it automatically creates a userData that uses cfn-init and cfn-signal scripts, assuming they have been installed on the instance.

#!/bin/bash
# fingerprint: 231cd699be373c15

(
    set +e
    /opt/aws/bin/cfn-init -v --region xxx --stack yyy --resource zzz -c default
    /opt/aws/bin/cfn-signal -e 0 --region xxx --stack yyy --resource zzz
    cat /var/log/cfn-init.log >&2
)

As @cprice404-aws comments, CDK does not allow adding commands to user data to install the cfn tools before using cfn-init, so the provision fails.

As a workaround, I have created an EC2 image builder pipeline to create an AMI with the Cfn helper scripts installed, and use that AMI to create the instance.

I attach the component to install cfn-* scripts:

name: CfnHelperScriptsDocument
description: This is CFN Helper Scripts document.
schemaVersion: 1.0

phases:
  - name: build
    steps:
      - name: CfnHelperScriptsStep
        action: ExecuteBash
        inputs:
          commands:
            - sudo apt update -y
            - sudo apt upgrade -y
            - sudo apt install python3-pip -y
            - pip3 install setuptools
            - sudo mkdir -p /opt/aws/bin
            - sudo wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz
            - sudo python3 -m easy_install --script-dir /opt/aws/bin aws-cfn-bootstrap-py3-latest.tar.gz
            - sudo ln -s /root/aws-cfn-bootstrap-py3-latest/init/ubuntu/cfn-hub /etc/init.d/cfn-hub
            - echo "Cfn Helper Scripts! Build."

NukaCody commented 3 years ago

Whenever the CDK is "too-opinionated" like in this case where it assumes the aws-cfn-bootstrap is already installed, my go-to are escape hatches. You can override user data like so

        const cfnInstance = instance.node.defaultChild as ec2.CfnInstance
        // const resourceLocator = `--region ${core.Aws.REGION} --stack ${core.Aws.STACK_NAME} --resource ${cfnInstance.logicalId}`;
        const resourceLocator = `--region ${core.Stack.of(this).region} --stack ${core.Stack.of(this).stackName} --region ${cfnInstance.logicalId}`
        cfnInstance.userData = core.Fn.base64(`#!/bin/bash -xe
sudo yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v ${resourceLocator} -c default
/opt/aws/bin/cfn-signal -e $? -v ${resourceLocator} -c default
cat /var/log/cfn-init.log >&2
`)

I tried both, core.Aws.REGION and core.Stack.of(this).region and the only difference I've found is the former makes a psuedo parameter AWS::REGION while the latter compiles to the actual region us-east-1. Not really sure the implications or trade-offs of using one way over the other.

If you're curious where I got some of the ideas, I stole them

But be warn, this assumes that your instance will be able to reach the internet, specifically the repo hosting aws-cfn-bootsrap. In my case, I needed a completely isolated subnet (no NAT) so I will have to look into frankisans solution of building a ready baked AMI and referencing that.

BDeus commented 3 years ago

Imho, assuming the location of cfn-init is a mistake. I use a custom AMI and cfn-init/signal is not install with a yum package (debian image) Python install with https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz install binary in /usr/local/bin At least we need to parameterized location or make the cfn-init script optionally.

mmuller88 commented 3 years ago

Whenever the CDK is "too-opinionated" like in this case where it assumes the aws-cfn-bootstrap is already installed, my go-to are escape hatches. You can override user data like so
        const cfnInstance = instance.node.defaultChild as ec2.CfnInstance
        // const resourceLocator = `--region ${core.Aws.REGION} --stack ${core.Aws.STACK_NAME} --resource ${cfnInstance.logicalId}`;
        const resourceLocator = `--region ${core.Stack.of(this).region} --stack ${core.Stack.of(this).stackName} --region ${cfnInstance.logicalId}`
        cfnInstance.userData = core.Fn.base64(`#!/bin/bash -xe
sudo yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v ${resourceLocator} -c default
/opt/aws/bin/cfn-signal -e $? -v ${resourceLocator} -c default
cat /var/log/cfn-init.log >&2
`)
I tried both, core.Aws.REGION and core.Stack.of(this).region and the only difference I've found is the former makes a psuedo parameter AWS::REGION while the latter compiles to the actual region us-east-1. Not really sure the implications or trade-offs of using one way over the other.

If you're curious where I got some of the ideas, I stole them

But be warn, this assumes that your instance will be able to reach the internet, specifically the repo hosting aws-cfn-bootsrap. In my case, I needed a completely isolated subnet (no NAT) so I will have to look into frankisans solution of building a ready baked AMI and referencing that.

you do have a typo and specified two time the region. but it should be:

`--region ${cdk.Stack.of(this).region} --stack ${cdk.Stack.of(this).stackName} --resource ${cfnInstance.logicalId}`;

mmuller88 commented 3 years ago

If someone struggle with that I've got it running with:

const cfnKaliInstance = kaliInstance.node.defaultChild as ec2.CfnInstance;
const resourceLocator = `--region ${cdk.Stack.of(this).region} --stack ${cdk.Stack.of(this).stackName} --resource ${cfnKaliInstance.logicalId}`;
cfnKaliInstance.cfnOptions.creationPolicy = {
  resourceSignal: {
    count: 1,
    timeout: cdk.Duration.minutes(15).toIsoString(),
  },
};
cfnKaliInstance.userData = cdk.Fn.base64(`#!/bin/bash -xe
... other setup stuff

apt-get install -y python-setuptools
mkdir -p /opt/aws/bin
wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
python -m easy_install --script-dir /opt/aws/bin aws-cfn-bootstrap-latest.tar.gz

/opt/aws/bin/cfn-init -v ${resourceLocator} -c default
/opt/aws/bin/cfn-signal -e 0 ${resourceLocator}
cat /var/log/cfn-init.log >&2
`);

felipeloha commented 3 years ago

+1

lschierer commented 2 years ago

I am struggling to get this to work with default ubuntu AMIs. An example of this in the cdk examples repo would be awesome.

shayaantx commented 2 years ago

So this hack worked for me on centos. Basically putting the installation of cfnbootstrap before the cfn-init commands like the other workarounds mentioned in this issue.


const autoScalingGroup = new as.AutoScalingGroup(appBuilder.stack, generateUniqueResourceId("auto-scaling-group"), {
...
}

const userDataScript = readFileSync('./install.sh', 'utf8');
autoScalingGroup.addUserData(userDataScript);
const preCfInitScript = [
    'sudo yum update -y',
    'sudo yum install -y epel-release',
    'sudo yum install -y python3 python3-pip',
    'sudo mkdir -p /opt/aws/bin',
    'sudo pip3 install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-latest.tar.gz',
    'sudo ln -s /usr/local/bin/cfn* /opt/aws/bin'
];
// @ts-ignore
autoScalingGroup.userData["lines"] = preCfInitScript.concat(autoScalingGroup.userData["lines"]);

maximillianus commented 2 years ago

Hi I happen to have similar issue too and I am on CDK v2. Basically init parameter is creating cfn-init and cfn-signal in UserData section. For AMI in which Cloudformation Helper scripts are preinstalled, this is fine however for AMI in which they aren't, that's the problem.

I can make a workaround by inserting UserData before the init-generated script or to add init after I manually created UserData, however does anyone know how to insert UserData or to add init after manual creation of UserData is done?

aws / aws-cdk