Documentation request - valid `userdata` examples

agile-jtdressel commented 2 years ago

As someone deploying default macOS AMIs, I'd like to see what sort of userdata I can provide via the AWS Web Console.

The Userdata Example shows how to include a Execute-User-Data module in my init.toml. I think this example would be useful if I were editing init.toml with the intent to make a reusable AMI.

I would like to configure some automatic tasks without creating a new AMI. I would not be changing init.toml, and would instead only vary the userdata. Examples would be helpful because the dedicated hosts have a 24 hour minimum (so I don't want to spin up multiple hosts to experiment) and since they take a while to wipe between runs.

My ideal solution would include:

[ ] Explicitly stating that you cannot use #cloud-config or an example showing how you can add. (I don't think the examples from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html work with ec2-macos-init)
[ ] Explicitly stating that you can use a shell script, and including a screenshot of doing so via the aws web console.
[ ] Explicitly stating that you can (or cannot) include an executable, and how it's run. (I'm assuming that the userdata is written to disk, given the +x flag, and then ec2-macos-init basically just invokes /path/to/userdata)

mattcataws commented 2 years ago

Hey @agile-jtdressel, thanks for opening an issue to bring these documentation gaps to our attention. I agree that the documentation and examples provided for the UserDataModule can be improved to help catch this misunderstanding before folks launch instances.

We'll look into making these changes and will keep this ticket updated along the way.

staranto commented 2 years ago

@mattcataws While you're working on the formal docs, can you provide here a simple Hello World example, to get us started? @agile-jtdressel is right on target - the lengthy lifecycle of these instances make random experimenting impossible.

As a trivial example - how can I do this with userdata and ec2-macos-init?

#! /bin/bash
whoami > /tmp/userdata.log

When I include that in the User Data on a Mac EC2 instance, I get no apparent results other than history.json showing that the '4_RunPerBoot' was successful. When I ec2-macos-init run I get --

2021/11/02 20:45:15.091269 Successfully completed module [ExecuteUserData] 
(type: userdata, group: 4) with message: provided user data is not executable 
(detected type: text/plain; charset=utf-8)

which is useless. A simple example would be so helpful!

mattcataws commented 2 years ago

Hey @staranto, thanks for following up on this issue. We're still working on updating our docs so I'll leave another comment when I have more news to share.

That example userdata you posted should work. Here's what it looked like on the instances I tested:

userdata set in the Console:

#!/bin/bash
whoami > /tmp/userdata.log

File generated on disk by userdata:

ec2-user ~ % cat /tmp/userdata.log
root

The log message that you saw comes up when the UserData module get's an "exec format error" while trying to execute the userdata. Could you verify the contents of the userdata file on your instance and try running it manually to see if you get a similar problem.

The path is - /usr/local/aws/ec2-macos-init/instances/<instance-id>/userdata

bpmct commented 2 years ago

Hi - are there any resources for this? I do not know how to use userdata with Mac EC2s

mattcataws commented 2 years ago

Hey @bpmct, the Userdata section of the README outlines how userdata is handled in EC2 Mac instances.

You can put a shell script in the userdata and ec2-macos-init will execute that script when the instance boots. You can also put non-executable data like JSON in the userdata to have it dropped on the instance.

Currently, the only resources available are the README and what's posted in this issue. We will be improving the resources provided to better outline how userdata can be used on EC2 macOS instances.

henryprecheur commented 2 years ago

I get the message:

Successfully completed module [ExecuteUserData] (type: userdata, group: 4) with message: provided user data is not executable (detected type: text/plain; charset=utf-8)

Unfortunately there’s no file named /usr/local/aws/ec2-macos-init/instances/<instance-id>/userdata although history.json is present. This is surprising since there’s no error message that the userdata file couldn’t be open...

I am planning to work around this issue by defining a new Command: https://github.com/aws/ec2-macos-init/blob/master/README.md#command That will download the userdata and execute it, but it would be great if this worked out of the box :-P FYI the AMI I used was ami-0dddc64bc72740146

mattcataws commented 2 years ago

Hey @henryprecheur, I'm sorry to hear that you're also experiencing issues with userdata. Could you please open a separate issue for this problem you're seeing. I'd like to keep this issue focused on the documentation request but I also want to investigate this problem.

In that new issue, could you include the userdata you're trying to execute. I'd also be interested to see what your work around Command looks like as well as the logs from when the command runs.

henryprecheur commented 2 years ago

The use data I’m trying to execute is something like:

#!/bin/sh
echo test > /tmp/test

It doesn’t matter how simple the script is: it never gets executed because the userdata file never gets created.

This isn’t a documentation issue: the user data execution feature is clearly broken, at least on the Big Sur image I tried and many other before. Reading this ticket: it is clear to me that most users know how to use the feature, it just doesn't work.

okudajun commented 2 years ago

Hello @henryprecheur,

Unfortunately, I couldn't reproduce your issue with the given script. As Matt requested, could you open a separate issue to focus on your issue? That'd be great if you share the following information when you open an issue. Please keep the user-data as simple as possible and please don't include sensitive data.

AMI ID you used.
Output of sudo /usr/local/bin/ec2-macos-init version.
The contents of the user-data after
How you launched instances with the user-data.
Output of aws ec2 describe-instance-attribute --instance-id <your instance id> --attribute userData

I'll leave some notes from my repro attempt. I used the given script below.

$ cat userdata.sh
#!/bin/sh
echo test > /tmp/test

I launched an instance with the following command line with the latest Big Sur AMI named amzn-ec2-macos-11.6.5-20220412-231843.

$ AMI="ami-04bf4e24478eb9533"
$ aws --region us-west-2 ec2 run-instances \
    --image-id $AMI \
    --metadata-options "HttpEndpoint=enabled,HttpTokens=required" \
    --instance-type mac1.metal \
    --subnet-id <one of my subnet> \
    --security-group-ids <one of my security group> \
    --key-name my_key \
    --iam-instance-profile Name="my_instance_profile" \
    --user-data file://userdata.sh

The launched instance had an attribute of an instance attribute of user-data of the given contents.

$ aws ec2 describe-instance-attribute --instance-id "i-06c96530ab1a60f8e" --attribute userData --query 'UserData.Value'
"IyEvYmluL3NoCmVjaG8gdGVzdCA+IC90bXAvdGVzdAo="

$ echo "IyEvYmluL3NoCmVjaG8gdGVzdCA+IC90bXAvdGVzdAo=" | base64 -d
#!/bin/sh
echo test > /tmp/test

After logging into the launched instance, I could see the following command results.

ec2-user@ip-172-31-56-152 ~ % last | grep boot
reboot    ~                         Tue May 17 05:21
ec2-user@ip-172-31-56-152 ~ % ls -l /tmp/test
-rw-r--r--  1 root  wheel  5 May 17 05:22 /tmp/test
ec2-user@ip-172-31-56-152 ~ % cat /tmp/test
test

The ec2-macos-init log includes the following logs.

ec2-user@ip-172-31-56-152 ~ % sudo grep ExecuteUserData /var/log/amazon/ec2/ec2-macos-init.log
2022/05/16 22:22:22.894362 Running module [ExecuteUserData] (type: userdata, group: 4)
2022/05/16 22:22:22.909480 Successfully completed module [ExecuteUserData] (type: userdata, group: 4) with message: successfully ran user data with stdout: [] and stderr: []

The user-data contents are stored under /usr/local/aws/ec2-macos-init/instances/<instance id>/userdata.

ec2-user@ip-172-31-56-152 ~ % cat /usr/local/aws/ec2-macos-init/instances/i-*/userdata
#!/bin/sh
echo test > /tmp/test

mschultz-aofl commented 2 years ago

I can confirm, I'm experiencing this same issue. I was directed to use ec2-macos-init by AWS support engineers. I see the same behavior as above, no userdata file is created and an error similar to Successfully completed module [ExecuteUserData] (type: userdata, group: 4) with message: provided user data is not executable (detected type: text/plain; charset=utf-8)

This is how I'm creating the instance within terraform:

resource "aws_instance" "mac_signing" {
  instance_type          = "mac1.metal"
  availability_zone      = aws_ec2_host.mac_signing.availability_zone
  host_id                = aws_ec2_host.mac_signing.id
  subnet_id              = data.aws_subnet.mac_subnet.id
  vpc_security_group_ids = [aws_security_group.mac_ssh.id]
  ami                    = data.aws_ami.mac.image_id
  key_name               = "aofl-ops"
  user_data_replace_on_change = true
  user_data=<<EOT
#!/bin/bash
echo Hi # Redacted for clarity
EOT
}

sudo /usr/local/bin/ec2-macos-init version

EC2 macOS Init
Version: 1.5.3 [2022-06-09 16:07:51 -0700]
https://github.com/aws/ec2-macos-init
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

AMI-ID: ami-029fbc1dc4f1f2192 Note: Above is a custom AMI generated by packer, built off of the latest big sur 11.6.6. This has several major changes:

Installed automation utils/xcode
Changed password to known password
Set up auto-login
Enable VNC support
autounlock of keychains

I can also confirm that the output of aws ec2 describe-instance-attribute --instance-id <your instance id> --attribute userData has the base64 encoded version of the userdata that was specified in the terraform. I can also view this userdata correctly thru the web console.

I believe this may be related to the changing of the password. Unfortunately due to MacOS being as terrible as it is, a random password isn't really feasible. Changing it after first-run also isn't, due to the hour+ it takes to reboot these instances and have the change take effect on e.g., keychains.

okudajun commented 1 year ago

Hello @mschultz-aofl,

Could you check what you would see under /usr/local/aws/ec2-macos-init/instances/<your instance ID>/. I think you'll see json.history and userdata. Then, could you try the following?

check the contents of userdata.
execute the userdata command manually.

My understanding is that the expected contents are the following.

#!/bin/bash
echo Hi # Redacted for clarity

On the other hand, ec2-macos-init failed the execution, and there might be something different.

ec2-macos-init will be run by root user, and I don't find any direct relationships between the ec2-user password change and the absence of execution. Let us check the contents of userdata and the result of manual execution of userdata first.

aws / ec2-macos-init

Documentation request - valid `userdata` examples #6