Support AL2023 & Bottlerocket for node groups

[flostadler]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

When creating a ManagedNodeGroup/NodeGroup with kubeletExtraArgs or bootstrapExtraArgs the component creates a LaunchTemplate/LaunchConfig with custom EC2 userdata to bootstrap the node. Right now this LaunchTemplate/LaunchConfig is hardcoded to use AL2. AL2 will be EOL in June 2025 and users need enough time to upgrade to newer AMIs (Bottlerocket or AL2023).

This change will involve changes to the way userdata is structured because Amazon Linux 2023 (AL2023) introduces a new node initialization process nodeadm that uses a YAML configuration schema and Bottlerocket structures user data in the TOML format (see AWS docs).

Necessary changes:

• [x] Remove AL2 hardcoding to start supporting AL2023 and Bottlerocket
  • [x] PR: https://github.com/pulumi/pulumi-eks/pull/1328
• [x] Support crafting custom user data
  • [x] AL2023
    • [x] PR: https://github.com/pulumi/pulumi-eks/pull/1332
  • [x] Bottlerocket
    • [x] PR: https://github.com/pulumi/pulumi-eks/pull/1334

Necessary bug fixes:

• [x] Fix self managed node groups wrongly assuming that nodes only have one block device
  • [x] PR: https://github.com/pulumi/pulumi-eks/pull/1337
• [x] ManagedNodeGroup with custom launch template always use x86 AMI even when graviton instances are used: #1323
• [x] eks.ManagedNodeGroup with amiType: 'AL2023_ARM_64_STANDARD' breaks when kubeletExtraArgs specified #1179
  • [x] PR: https://github.com/pulumi/pulumi-eks/pull/1328

Affected area/feature

• pulumi-eks

[flostadler]

I found an issue with ManagedNodeGroup not properly handling instances with ARM64 architecture. Need to fix this first: #1323

[flostadler]

1323 is fixed now.

Continuing with this work now.

In order for ami type selection to work we'll also need to fix https://github.com/pulumi/pulumi-eks/issues/1179.

[flostadler]

The feature branch has support for AL2023 & Bottlerocket now: https://github.com/pulumi/pulumi-eks/tree/flostadler/al2023-bottlerocket-support.

Closing this out while we work on hardening this before we move it to the main branch