Run commands at the end of bootstrapping (AL2023)

[dntosas]

Description

Hello team!

Started testing AL2023 with the option instanceStorePolicy: RAID0. While this option creates as expected the mount /mnt/k8s-disks/0/ , this seems to be create after running the provided userData. As a result, we can't run commands against this path until created. Any ideas how to overcome this or merge it in appropriate order?

I guess we could mitigate this by putting the actions for mounting this volume on top of merged userData file?

Example config with storePolicy + userData:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: test
spec:
  amiFamily: AL2023
  instanceProfile: XXX
  instanceStorePolicy: RAID0
  userData: >
    MIME-Version: 1.0

    Content-Type: multipart/mixed; boundary="==EC2INIT=="

    --==EC2INIT==

    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/bin/bash
    NVM_PATH="/mnt/k8s-disks/0"
    if [[ -d "$NVM_PATH" ]]; then
      mkdir $NVM_PATH/test_directory
    fi
    --==EC2INIT==

Observed Behavior: mount /mnt/k8s-disks/0/ is created after the userData script is running thus we can't utilize it for running arbitrary commands.

Expected Behavior: userData script to create directory on the new mounted RAID0 volume

Reproduction Steps (Please include YAML): attached above ^

Versions:

• Chart Version: 0.35.4
• Kubernetes Version (kubectl version): 1.28

[dntosas]

this is running as i understand when RAID0 is given as storePolicy --> https://github.com/awslabs/amazon-eks-ami/blob/761c8d2f4a4e964010ea162bac841de27b9f1a65/templates/shared/runtime/bin/setup-local-disks#L33

[njtran]

This is not specific to the RAID0 feature, but specifically with our bootstrapping logic. What sort of commands are you running that you need to run as part of the bootstrapping process? Seems like you just need to create a new directory, which seems like you can do that as soon as the instance is booted and the volumes are mounted.

[jmdeal]

I don't think this in particular has to do with our bootstrapping logic, but rather nodeadm's. It looks like nodeadm doesn't run the setup-local-disks script responsible for configuring and mounting the NVME drives until after the cloud init scripts have run (ref). This wouldn't be affected by how Karpenter orders the user data. I'd recommend opening a feature request over in the EKS AMI repo.