When creating an EC2 instance from a derived AWS AMI, authorized_keys is not updated from launch configuration

[kurtschelfthout]

I have followed the steps in https://docs.01.org/clearlinux/latest/get-started/cloud-install/import-clr-aws.html to create a base Clear Linux AMI. This base image works fine - I can create an instance from it and log in using any SSH key pair I configure at instance launch.

However, when I take another AMI from that instance, without doing anything else with it except logging in, when I start an EC2 instance from the derived AMI it does not pick up the ssh key pair I configure at launch of the EC2 instance. ~/.ssh/authorized_keys does not seem to be modified. I can still log in to the derived instance by using the original key. If I do that, and run ucd-data-fetch manually, it does work and adds the configured key to authorized_keys.

To be abundantly clear - steps:

1. Create a base Clear Linux AMI which I'll call BASE AMI as in https://docs.01.org/clearlinux/latest/get-started/cloud-install/import-clr-aws.html. I'm pretty sure this also reproduces with the AWS marketplace AMI.
2. Launch an EC2 instance from BASE AMI with ssh key pair BASE SSH KEY PAIR.
3. Log into the instance with BASE SSH KEY PAIR. Everything works. ~.ssh/authorized_keys contains BASE SSH KEY PAIR.
4. Create an AMI from this EC2 instance, which I'll call DERIVED AMI.
5. Launch a new EC2 instance from DERIVED AMI with a new ssh key pair DERIVED SSH KEY PAIR
6. Try to log into the new instance using DERIVED KEY PAIR. This fails.
7. Log into the new instance using BASE KEY PAIR. This works!
8. ~/.ssh/authorized_keys contains only BASE KEY PAIR, and not DERIVED KEY PAIR
9. run sudo ucd-data-fetch aws
10. ~/.ssh/authorized_keys now contains BASE KEY PAIR, and DERIVED KEY PAIR

So it seems like somehow at startup of the instance started from the derived AMI, the AWS configuration is not working or not applied correctly.

[kurtschelfthout]

Meanwhile found out that the service ucd-aws which calls ucd-data-fetch only runs if /var/lib/cloud/aws-user-data does not exist. So before an AMI is taken, this file needs to be deleted, or the user data won't be generated.

However is that I ought to be able to delete ~/.ssh/authorized_keys and the server keys in /etc/ssh as well, but if I do either of those things it seems like they are not being re-generated, despite that /var/lib/cloud/aws-user-data is deleted, and definitely re-generated. Also on startup the new key is added to ~/.ssh/authorized_keys, but if I delete it and take an AMI I still can't log in.

[bryteise]

@mdpitterle any advice on this perhaps?

[kurtschelfthout]

Thanks in advance for any help - I finally realized there's a much simpler way to put the problem:

How can I take the standard AWS Clear Linux image, customize it (in my case add docker and python, enable sftp) and then make a new, usable AMI from that?

The use case here is to have a customized AMI available for users and right now the customized Clear AMI does not pick up ssh keys robustly.

[kurtschelfthout]

I think I have it figured out now - run this before taking a snapshot/AMI and it works:

sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
shred -u ~/.ssh/authorized_keys
sudo shred -u /var/lib/cloud/aws-user-data

sudo shred -u /etc/machine-id 

The first three lines are ssh related (earlier I forgot a -u argument to shred sigh). The last is so that Clear generates a new id, e.g. for uname -n. Not strictly necessary, but weird if all machines have the same name.

Up to you what you want to do with this - I guess from my perspective it would be nice if this was at least documented somehwere, and also I'm not sure I haven't forgotten anything.