Unable to login to ec2 instance when `user data` is provided

[shivarammysore]

Describe the bug

Create a ec2 instance on AWS with latest Fedora CoreOS stable stream. Use a preconfigured SSH key on AWS. Leave user data empty. We can ssh into that instance with the key.

aws ec2 run-instances \
 --region us-east-1 \
 --image-id ami-074ffe616eec21f4f \
 --count 1 \
 --instance-type t2.micro \
 --key-name myawskey \
 --security-group-ids sg-e7e38e4b2 \
 --subnet-id subnet-26373 \
 --associate-public-ip-address \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=name,Value=test-ami}, {Key=cost-center,Value=RandD}]' \
 --query 'Instances[0].InstanceId'

With the above created instance, we can ssh into the instance

Now, include user data

aws ec2 run-instances \
 --region us-east-1 \
 --image-id ami-074ffe616eec21f4f \
 --count 1 \
 --instance-type t2.micro \
 --key-name myawskey \
 --security-group-ids sg-e7e38e4b2 \
 --subnet-id subnet-26373 \
 --associate-public-ip-address \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=name,Value=test-ami}, {Key=cost-center,Value=RandD}]' \
--user-data file:///Users/smysore/aws_fcos_config.bu \
 --query 'Instances[0].InstanceId'

{
  "ignition": {
    "version": "3.3.0"
  },
  "storage": {
    "files": [
      {
        "path": "/etc/selinux/config",
        "contents": {
          "compression": "",
          "source": "data:,SELINUX%3Ddisabled%0ASELINUXTYPE%3Dtargeted%0A"
        },
        "mode": 272
      },
      {
        "path": "/etc/chrony.conf",
        "contents": {
          "compression": "gzip",
          "source": "data:;base64,H4sIAAAAAAAC/2RTwY7bRgy96ysI+JIUyGjttIumt8WiKHIpgsSLnukRJXE9GqoktYb364sZaVMHOdnQ8L15897jDh66Dh7++QZ/H7/AN9IXUnABHwlcZpC+/k1s3th6ur//FA6//RrK7/7wEfi0qDnMSj1p0+zgyQjm5ZQ4wgox6FWmSjSLpJB9DqIDzCrPFD00O/iSCI0gSjbuSOFZOHMevmPg3eg+/9G2l8sl3JK0ZTKMPqX3oamTh9BTJ4o/jG0qi7yvFEW7yqzoBOhwGTmO9YtdzWmCmCSeYUDO1iYxIwPniULTKffecyJoX1DbxKc2jir52taTwv+Qklx+JnOBE4E5zTN1wLlO9Fys81GJYJk7dLJmB9wDu4H0vZEDGyTUoeQyYoY9GEXJXWgmPFPhg324g4/l6j8znhLBmTRTArvmqo1f0VnyW5ZKmD6U12zC3n09Pr4PjXosgBuaEbW7oFJ9ujlOc0lEMmBKwNlJe4zFmREdbJlnUQcucY6X7xD4pTB+zlFrwEXAxJmnZYK8TCfSIssoUfR6qcmihVTp34WVuuIbds+LebP7ydTQ7CbOb5DD/+6XMsfElB0wRrKtgUkiJsjkF9ECxjq8/3QI+/vfw124a/f3haTuQX020AvlEkgWvzF007WOrPeHZrfSmyv6MsH+rlLNFLm/Qu1MlOy4FvtMV4NetErFxUfKzrEGFZozXdeOkcetX6EACuFf5HB8+Pzh6fj41hDMHSTCeWvGzbptZvkrdOh4QqPQlEmj6K+gPIzePh0fb4V2rBRd9FrVJRmqdAtNkqFj3Xovw6ZrtavEt20R5150WhtXqivDQF01Z4CJ0BalibIbmKOzOUcDV4xnzkPzXwAAAP//nZoSr48EAAA="
        },
        "mode": 272
      },
      {
        "overwrite": false,
        "path": "/etc/fedora-coreos-pinger/config.d/99-disable-reporting.toml",
        "contents": {
          "source": "data:text/plain;charset=iso-8859-7,%5Breporting%5D%0Aenabled%20%3D%20false"
        },
        "mode": 420
      },
      {
        "overwrite": false,
        "path": "/etc/zincati/config.d/10-enable-feature.toml",
        "contents": {
          "source": "data:text/plain;charset=iso-8859-7,%5Bfeature%5D%0Aenabled%20%3D%20true"
        },
        "mode": 420
      }
    ]
  }
}

When instance created with the above command, we can ssh login to the instance. I am not sure what the issue is.

Reproduction steps

Describe in the bug description above

Expected behavior

Should be able to login with user data provided.

Actual behavior

Describe in the bug description above

System details

AWS EC2 t2.micro

IMAGE_DESC="Fedora CoreOS stable 37.20221106.3.0 x86_64" IMAGE_ID='ami-074ffe616eec21f4f'

Ignition config

{
  "ignition": {
    "version": "3.3.0"
  },
  "storage": {
    "files": [
      {
        "path": "/etc/selinux/config",
        "contents": {
          "compression": "",
          "source": "data:,SELINUX%3Ddisabled%0ASELINUXTYPE%3Dtargeted%0A"
        },
        "mode": 272
      },
      {
        "path": "/etc/chrony.conf",
        "contents": {
          "compression": "gzip",
          "source": "data:;base64,H4sIAAAAAAAC/2RTwY7bRgy96ysI+JIUyGjttIumt8WiKHIpgsSLnukRJXE9GqoktYb364sZaVMHOdnQ8L15897jDh66Dh7++QZ/H7/AN9IXUnABHwlcZpC+/k1s3th6ur//FA6//RrK7/7wEfi0qDnMSj1p0+zgyQjm5ZQ4wgox6FWmSjSLpJB9DqIDzCrPFD00O/iSCI0gSjbuSOFZOHMevmPg3eg+/9G2l8sl3JK0ZTKMPqX3oamTh9BTJ4o/jG0qi7yvFEW7yqzoBOhwGTmO9YtdzWmCmCSeYUDO1iYxIwPniULTKffecyJoX1DbxKc2jir52taTwv+Qklx+JnOBE4E5zTN1wLlO9Fys81GJYJk7dLJmB9wDu4H0vZEDGyTUoeQyYoY9GEXJXWgmPFPhg324g4/l6j8znhLBmTRTArvmqo1f0VnyW5ZKmD6U12zC3n09Pr4PjXosgBuaEbW7oFJ9ujlOc0lEMmBKwNlJe4zFmREdbJlnUQcucY6X7xD4pTB+zlFrwEXAxJmnZYK8TCfSIssoUfR6qcmihVTp34WVuuIbds+LebP7ydTQ7CbOb5DD/+6XMsfElB0wRrKtgUkiJsjkF9ECxjq8/3QI+/vfw124a/f3haTuQX020AvlEkgWvzF007WOrPeHZrfSmyv6MsH+rlLNFLm/Qu1MlOy4FvtMV4NetErFxUfKzrEGFZozXdeOkcetX6EACuFf5HB8+Pzh6fj41hDMHSTCeWvGzbptZvkrdOh4QqPQlEmj6K+gPIzePh0fb4V2rBRd9FrVJRmqdAtNkqFj3Xovw6ZrtavEt20R5150WhtXqivDQF01Z4CJ0BalibIbmKOzOUcDV4xnzkPzXwAAAP//nZoSr48EAAA="
        },
        "mode": 272
      },
      {
        "overwrite": false,
        "path": "/etc/fedora-coreos-pinger/config.d/99-disable-reporting.toml",
        "contents": {
          "source": "data:text/plain;charset=iso-8859-7,%5Breporting%5D%0Aenabled%20%3D%20false"
        },
        "mode": 420
      },
      {
        "overwrite": false,
        "path": "/etc/zincati/config.d/10-enable-feature.toml",
        "contents": {
          "source": "data:text/plain;charset=iso-8859-7,%5Bfeature%5D%0Aenabled%20%3D%20true"
        },
        "mode": 420
      }
    ]
  }
}

Additional information

No response

[miabbott]

It looks like your Ignition config is failing to be applied because you are trying to write out /etc/chrony.conf, but you don't have the overwrite option provided:

[ 43.873286] ignition[918]: CRITICAL : Ignition failed: failed to create files: failed to create files: error creating /sysroot/etc/chrony.conf: error creating file "/sysroot/etc/chrony.conf": A file exists there already and overwrite is false

You can cheaply verify your Ignition configs by using coreos-assembler run -i <path/to/ignition.config> to see if a local VM will boot using your config.

[shivarammysore]

Thanks @miabbott . Issue is resolved. Thanks for the tip on using coreos-assembler. Will add that to my notes.