AWS CloudFormation stack fails with "UnicodeDecodeError"

multimeric commented 5 years ago

I have been attempting to spin up the CloudFormation stack provided here. This intermittently fails and succeeds, I'm not sure why it sometimes does and sometimes doesn't work.

When it fails, it's because the EC2 instance fails to initialize. The most promising section I can find from the EC2 logs shows the following:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 891, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 49: ordinal not in range(128)
Logged from file util.py, line 476
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 891, in emit
    stream.write(fs % msg.encode("UTF-8"))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 71: ordinal not in range(128)
Logged from file util.py, line 476
[   72.975338] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
Error occurred during build: Command 04InstallECSAdditions failed

The line EXT4-fs (dm-3): mounted filesystem with ordered data mode is repeated about 100 times in the real logs (below), I've just abridged it here for clarity. Anyway, the main thing this tells us that it's failing during step 04 of the EC2 startup script, which does the following:

04InstallECSAdditions:
  command:
    Fn::If:
      - UseCromwell
      - !Join [" ", ["sh", "/opt/ecs-additions/ecs-additions-cromwell.sh"]]
      - echo "OK"
  env:
    PATH: "/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin"

My best guess as to what is happening, is this blog post, which suggests:

Apparently cfn-init has a limit on the amount of output it can process from a command, and I was pushing that limit.

I suspect the reason for the UTF8 error is that the output was truncated between two bytes or something, and when the parser underneath cfn-init tried to parse it, it encountered what appeared to be an invalid UTF8 character.

So perhaps the reason this issue is intermittent is because the length of the logs from this command are occasionally too long for the cfn-init script? Or this might be a red herring.

To aid with debugging, here are some useful logs

ec2_log.txt: This is the console output from the EC2 instance that failed. I've censored out some of the key data, just in case any of it involves my own public key (probably unnecessary, but I doubt it's related)
stack_description.json.txt: The output from aws cloudformation describe-stacks on the stack I spun up. This should give you the exact parameters I used when it last failed.