search

eksctl-io / eksctl

The official CLI for Amazon EKS
https://eksctl.io
Other
4.92k stars 1.41k forks source link

[Help] Unable to create EKS managed nodegroups with custom Ubuntu AMI #6491

Closed shivkumr closed 1 year ago

shivkumr commented 1 year ago 
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: private-eks-cluster
  region: us-east-1
  version: "1.24"

vpc:
  subnets:
    private:
      us-east-1a: 
        id: subnet-xxxxxx
      us-east-1b: 
        id: subnet-xxxxxxx

managedNodeGroups:
  - name:  worker
    labels: { nodeType: worker }
    instanceType: t2.xlarge
    desiredCapacity: 2
    ami: "ami-003237eb446483b5"
    privateNetworking: true
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh private-eks-cluster

$ eksctl create cluster -f ./cluster.yaml Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag

Trying to create a eks cluster using the ubuntu custom AMI from canonical, eksctl is complaining to set amiFamily, but when I add amiFamily then the request fails with the below error. 

$ eksctl create cluster -f ./cie-cluster-custom-ubuntu.yaml 
Error: cannot set amiFamily to Ubuntu2004 when using a custom AMI for managed nodes, only AmazonLinux2 is supported
github-actions[bot] commented 1 year ago

Hello shivkumr :wave: Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

Sathyam-Unicommerce commented 1 year ago

HI All, we are facing the same issue as well. I created a custom ubuntu 20.04 ami. Like @shivkumr mentioned, it expects an amiFamily field in the config file but denies any value except AmazonLinux2.

I have referred to the documentation available at https://eksctl.io/usage/custom-ami-support/ and have noticed that it does not mention any restrictions on using a custom Ubuntu AMI for creating a managed nodegroup. Furthermore, there are examples available on this page that demonstrate the creation of managed nodegroups using custom AMIs, which gives me the impression that using a custom Ubuntu AMI to create managed nodegroups should be possible.

I then tried creating an unmanaged nodegroup and although it was successfully created via eksctl, it did not get listed on the EKS console. However, I was able to run some pods on the unmanaged nodegroup, which worked as expected. I can see the nodes and pods in the workload section on eks console.

Himangini commented 1 year ago

@Sathyam-Unicommerce you're right, the docs example mentioned at https://eksctl.io/usage/custom-ami-support/ was added 2 years ago and I suspect this page info is outdated and needs work. We'll investigate this issue further 

nodeGroups:
  - name: ng1
    instanceType: m5.large
    amiFamily: AmazonLinux2
managedNodeGroups:
  - name: m-ng-2
    instanceType: m5.large
    amiFamily: Ubuntu2004
jicowan commented 1 year ago

I am having a similar issue. When I specify an AMI in my configuration file, it complains that I need to explicitly set the amiFamily. When I set the amiFamily, it says I need to add the overrideBootstrapCommand.

cluster.yaml

--- 
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: custom-vpc
  region: us-east-2
  version: "1.26"

vpc:
  id: "vpc-0540b5bb2f62f0b57"  # (optional, must match VPC ID used for each subnet below)
  cidr: "10.0.0.0/16"       # (optional, must match CIDR used by the given VPC)
  subnets:
    # must provide 'private' and/or 'public' subnets by availability zone as shown
    private:
      us-east-2a:
        id: "subnet-055f3a6c3ff813545"
        cidr: "10.0.128.0/20" # (optional, must match CIDR used by the given subnet)

      us-east-2b: 
        id: "subnet-01d641621d904f194"
        cidr: "10.0.144.0/20"

    public:
      us-east-2a: 
        id: "subnet-02c70740e6bcca41c"
        cidr: "10.0.0.0/20"

      us-east-2b: 
        id: "subnet-0f05d1763e92d0854"
        cidr: "10.0.16.0/20"

nodeGroups:
  - name: ng-1
    instanceType: m5.xlarge
    desiredCapacity: 2
    privateNetworking: true # if only 'Private' subnets are given, this must be enabled
    # amiFamily: AmazonLinux2
    # ami: ami-07a45fc1d4fd1092a
    #  overrideBootstrapCommand: 

managedNodeGroups:
  - name: ng-2
    instanceType: m5.xlarge
    desiredCapacity: 2
    privateNetworking: true

first error

Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag

second error

Error: nodeGroups[0].overrideBootstrapCommand is required when using a custom AMI (nodeGroups[0].ami)
OneOfTheJohns commented 1 year ago

Any fixes yet?

TiberiuGC commented 1 year ago

I am having a similar issue. When I specify an AMI in my configuration file, it complains that I need to explicitly set the amiFamily. When I set the amiFamily, it says I need to add the overrideBootstrapCommand.

cluster.yaml

--- 
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: custom-vpc
  region: us-east-2
  version: "1.26"

vpc:
  id: "vpc-0540b5bb2f62f0b57"  # (optional, must match VPC ID used for each subnet below)
  cidr: "10.0.0.0/16"       # (optional, must match CIDR used by the given VPC)
  subnets:
    # must provide 'private' and/or 'public' subnets by availability zone as shown
    private:
      us-east-2a:
        id: "subnet-055f3a6c3ff813545"
        cidr: "10.0.128.0/20" # (optional, must match CIDR used by the given subnet)

      us-east-2b: 
        id: "subnet-01d641621d904f194"
        cidr: "10.0.144.0/20"

    public:
      us-east-2a: 
        id: "subnet-02c70740e6bcca41c"
        cidr: "10.0.0.0/20"

      us-east-2b: 
        id: "subnet-0f05d1763e92d0854"
        cidr: "10.0.16.0/20"

nodeGroups:
  - name: ng-1
    instanceType: m5.xlarge
    desiredCapacity: 2
    privateNetworking: true # if only 'Private' subnets are given, this must be enabled
    # amiFamily: AmazonLinux2
    # ami: ami-07a45fc1d4fd1092a
    #  overrideBootstrapCommand: 

managedNodeGroups:
  - name: ng-2
    instanceType: m5.xlarge
    desiredCapacity: 2
    privateNetworking: true

first error

Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag

second error

Error: nodeGroups[0].overrideBootstrapCommand is required when using a custom AMI (nodeGroups[0].ami)

@jicowan the errors you are seeing are appropriate for your use case. When using a custom AMI, you need to explicitly set AMI Family because eksctl does certain validations that wouldn't be possible without knowing which AMI Family you intend to use. Check out this PR for further reference.

The overrideBootstrapCommand field is also required when working with custom AL2 or Ubuntu AMIs, whereas for Bottlerocket and Windows is not supported. I admit the error can be more explicit by mentioning AL2 / Ubuntu AMI usage.

TiberiuGC commented 1 year ago

Hi @shivkumr , @Sathyam-Unicommerce as stated in the message above, the amiFamily field is now required to make sure eksctl properly performs all intended validations. This is a rather recent change and turns out it unveils this interesting validation that you're running into.

Error: cannot set amiFamily to Ubuntu2004 when using a custom AMI for managed nodes, only AmazonLinux2 is supported

I suspect in the past you could just use Ubuntu custom AMIs on EKS managed nodes by simply not specifying the amiFamily, essentially by-passing this validation. Before deciding whether to relax this validation of only supporting AL2 images, I wanted to ask you a couple questions:

bradwatsonaws commented 1 year ago

I am running into the same issue. I am trying to deploy a managed nodegroup of custom Red Hat Enterprise Linux worker nodegroups. I can work around this by first creating a simple launch template with the AMI ID specified and User Data specified to execute the bootstrap script, but it would be better to simply create the node group with the custom AMI ID and overrideBootstrapCommand specified.

TiberiuGC commented 1 year ago

Hi @shivkumr , @Sathyam-Unicommerce as stated in the message above, the amiFamily field is now required to make sure eksctl properly performs all intended validations. This is a rather recent change and turns out it unveils this interesting validation that you're running into.

Error: cannot set amiFamily to Ubuntu2004 when using a custom AMI for managed nodes, only AmazonLinux2 is supported

I suspect in the past you could just use Ubuntu custom AMIs on EKS managed nodes by simply not specifying the amiFamily, essentially by-passing this validation. Before deciding whether to relax this validation of only supporting AL2 images, I wanted to ask you a couple questions:

  • is your particular use case something that can be achieved through custom AMIs only, would the default Ubuntu AMI satisfy it?
  • provided in the past you used to be able to deploy EKS managed nodes with custom Ubuntu AMI, have you ever faced any bugs / limitations in features compared to AL2 images?

@shivkumr , @Sathyam-Unicommerce - waiting for some feedback on the above.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

roni-admon commented 1 year ago

joining to this thread. answering your question - my particular use case can be achieved through custom AMIs only...

defyjoy commented 1 year ago

I have the eksctl version 0.161.0 and the problem is still persisting. 

Error: could not create cluster provider from options: managedNodeGroups[0].overrideBootstrapCommand is required 
when using a custom AMI based on Ubuntu2004 (managedNodeGroups[0].ami)

Why was this closed and with what resolution ? Also , I needed to add amiFamily too.

image

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: EKS-SANDBOX-0
  region: eu-west-1
  version: "1.28"

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"
  - "logs"

vpc:
  id: vpc123
  securityGroup: sg-123
  sharedNodeSecurityGroup: sg-123
  manageSharedNodeSecurityGroupRules: true
  subnets:
    private:
      eu-west-1a:
        id: subnet-123
      eu-west-1b:
        id: subnet-123
      eu-west-1c:
        id: subnet-123

iam:
  withOIDC: true

managedNodeGroups:
- name: EKS-NG-0
  instanceType: t3.medium
  # ami: ami-0a9b2251badf0cf47
  # securityGroups:
  #   attachIDs: ["sg-1234"]
  minSize: 1
  desiredCapacity: 3
  maxSize: 5
  # amiFamily: AmazonLinux2 
  amiFamily: Ubuntu2004
  ami: ami-008b919c7799c96e5
  tags:
    nodegroup-type: Ubuntu2004 
  maxPodsPerNode: 110
  privateNetworking: true
  ssh:
    allow: false
  volumeSize: 100
  volumeName: /dev/xvda
  volumeEncrypted: true
  disableIMDSv1: true
  taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoSchedule"

cloudWatch:
  clusterLogging:
    enableTypes: ["all"]
    logRetentionInDays: 7
bradwatsonaws commented 1 year ago

I have the eksctl version 0.161.0 and the problem is still persisting.

Error: could not create cluster provider from options: managedNodeGroups[0].overrideBootstrapCommand is required 
when using a custom AMI based on Ubuntu2004 (managedNodeGroups[0].ami)

Why was this closed and with what resolution ? Also , I needed to add amiFamily too.

image

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: EKS-SANDBOX-0
  region: eu-west-1
  version: "1.28"

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"
  - "logs"

vpc:
  id: vpc123
  securityGroup: sg-123
  sharedNodeSecurityGroup: sg-123
  manageSharedNodeSecurityGroupRules: true
  subnets:
    private:
      eu-west-1a:
        id: subnet-123
      eu-west-1b:
        id: subnet-123
      eu-west-1c:
        id: subnet-123

iam:
  withOIDC: true

managedNodeGroups:
- name: EKS-NG-0
  instanceType: t3.medium
  # ami: ami-0a9b2251badf0cf47
  # securityGroups:
  #   attachIDs: ["sg-1234"]
  minSize: 1
  desiredCapacity: 3
  maxSize: 5
  # amiFamily: AmazonLinux2 
  amiFamily: Ubuntu2004
  ami: ami-008b919c7799c96e5
  tags:
    nodegroup-type: Ubuntu2004 
  maxPodsPerNode: 110
  privateNetworking: true
  ssh:
    allow: false
  volumeSize: 100
  volumeName: /dev/xvda
  volumeEncrypted: true
  disableIMDSv1: true
  taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoSchedule"

cloudWatch:
  clusterLogging:
    enableTypes: ["all"]
    logRetentionInDays: 7

Your issue is different than the issue that was closed. Your issue is being caused because when you specify an AMI, you must also include the overrideBootstrapCommand option in your yaml file.

See the documentation here: https://eksctl.io/usage/custom-ami-support/ And announcement for this requirement: https://eksctl.io/announcements/nodegroup-override-announcement/