Closed shivkumr closed 1 year ago
HI All, we are facing the same issue as well. I created a custom ubuntu 20.04 ami. Like @shivkumr mentioned, it expects an amiFamily field in the config file but denies any value except AmazonLinux2.
I have referred to the documentation available at https://eksctl.io/usage/custom-ami-support/ and have noticed that it does not mention any restrictions on using a custom Ubuntu AMI for creating a managed nodegroup. Furthermore, there are examples available on this page that demonstrate the creation of managed nodegroups using custom AMIs, which gives me the impression that using a custom Ubuntu AMI to create managed nodegroups should be possible.
I then tried creating an unmanaged nodegroup and although it was successfully created via eksctl, it did not get listed on the EKS console. However, I was able to run some pods on the unmanaged nodegroup, which worked as expected. I can see the nodes and pods in the workload section on eks console.
@Sathyam-Unicommerce you're right, the docs example mentioned at https://eksctl.io/usage/custom-ami-support/ was added 2 years ago and I suspect this page info is outdated and needs work. We'll investigate this issue further
nodeGroups:
- name: ng1
instanceType: m5.large
amiFamily: AmazonLinux2
managedNodeGroups:
- name: m-ng-2
instanceType: m5.large
amiFamily: Ubuntu2004
I am having a similar issue. When I specify an AMI in my configuration file, it complains that I need to explicitly set the amiFamily. When I set the amiFamily, it says I need to add the overrideBootstrapCommand.
cluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: custom-vpc
region: us-east-2
version: "1.26"
vpc:
id: "vpc-0540b5bb2f62f0b57" # (optional, must match VPC ID used for each subnet below)
cidr: "10.0.0.0/16" # (optional, must match CIDR used by the given VPC)
subnets:
# must provide 'private' and/or 'public' subnets by availability zone as shown
private:
us-east-2a:
id: "subnet-055f3a6c3ff813545"
cidr: "10.0.128.0/20" # (optional, must match CIDR used by the given subnet)
us-east-2b:
id: "subnet-01d641621d904f194"
cidr: "10.0.144.0/20"
public:
us-east-2a:
id: "subnet-02c70740e6bcca41c"
cidr: "10.0.0.0/20"
us-east-2b:
id: "subnet-0f05d1763e92d0854"
cidr: "10.0.16.0/20"
nodeGroups:
- name: ng-1
instanceType: m5.xlarge
desiredCapacity: 2
privateNetworking: true # if only 'Private' subnets are given, this must be enabled
# amiFamily: AmazonLinux2
# ami: ami-07a45fc1d4fd1092a
# overrideBootstrapCommand:
managedNodeGroups:
- name: ng-2
instanceType: m5.xlarge
desiredCapacity: 2
privateNetworking: true
first error
Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag
second error
Error: nodeGroups[0].overrideBootstrapCommand is required when using a custom AMI (nodeGroups[0].ami)
Any fixes yet?
I am having a similar issue. When I specify an AMI in my configuration file, it complains that I need to explicitly set the amiFamily. When I set the amiFamily, it says I need to add the overrideBootstrapCommand.
cluster.yaml
--- apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: custom-vpc region: us-east-2 version: "1.26" vpc: id: "vpc-0540b5bb2f62f0b57" # (optional, must match VPC ID used for each subnet below) cidr: "10.0.0.0/16" # (optional, must match CIDR used by the given VPC) subnets: # must provide 'private' and/or 'public' subnets by availability zone as shown private: us-east-2a: id: "subnet-055f3a6c3ff813545" cidr: "10.0.128.0/20" # (optional, must match CIDR used by the given subnet) us-east-2b: id: "subnet-01d641621d904f194" cidr: "10.0.144.0/20" public: us-east-2a: id: "subnet-02c70740e6bcca41c" cidr: "10.0.0.0/20" us-east-2b: id: "subnet-0f05d1763e92d0854" cidr: "10.0.16.0/20" nodeGroups: - name: ng-1 instanceType: m5.xlarge desiredCapacity: 2 privateNetworking: true # if only 'Private' subnets are given, this must be enabled # amiFamily: AmazonLinux2 # ami: ami-07a45fc1d4fd1092a # overrideBootstrapCommand: managedNodeGroups: - name: ng-2 instanceType: m5.xlarge desiredCapacity: 2 privateNetworking: true
first error
Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag
second error
Error: nodeGroups[0].overrideBootstrapCommand is required when using a custom AMI (nodeGroups[0].ami)
@jicowan the errors you are seeing are appropriate for your use case. When using a custom AMI, you need to explicitly set AMI Family because eksctl
does certain validations that wouldn't be possible without knowing which AMI Family you intend to use. Check out this PR for further reference.
The overrideBootstrapCommand
field is also required when working with custom AL2 or Ubuntu AMIs, whereas for Bottlerocket and Windows is not supported. I admit the error can be more explicit by mentioning AL2 / Ubuntu AMI usage.
Hi @shivkumr , @Sathyam-Unicommerce as stated in the message above, the amiFamily
field is now required to make sure eksctl
properly performs all intended validations. This is a rather recent change and turns out it unveils this interesting validation that you're running into.
Error: cannot set amiFamily to Ubuntu2004 when using a custom AMI for managed nodes, only AmazonLinux2 is supported
I suspect in the past you could just use Ubuntu custom AMIs on EKS managed nodes by simply not specifying the amiFamily
, essentially by-passing this validation. Before deciding whether to relax this validation of only supporting AL2 images, I wanted to ask you a couple questions:
is your particular use case something that can be achieved through custom AMIs only, would the default Ubuntu AMI satisfy it?
provided in the past you used to be able to deploy EKS managed nodes with custom Ubuntu AMI, have you ever faced any bugs / limitations in features compared to AL2 images?
I am running into the same issue. I am trying to deploy a managed nodegroup of custom Red Hat Enterprise Linux worker nodegroups. I can work around this by first creating a simple launch template with the AMI ID specified and User Data specified to execute the bootstrap script, but it would be better to simply create the node group with the custom AMI ID and overrideBootstrapCommand
specified.
Hi @shivkumr , @Sathyam-Unicommerce as stated in the message above, the
amiFamily
field is now required to make sureeksctl
properly performs all intended validations. This is a rather recent change and turns out it unveils this interesting validation that you're running into.
Error: cannot set amiFamily to Ubuntu2004 when using a custom AMI for managed nodes, only AmazonLinux2 is supported
I suspect in the past you could just use Ubuntu custom AMIs on EKS managed nodes by simply not specifying the
amiFamily
, essentially by-passing this validation. Before deciding whether to relax this validation of only supporting AL2 images, I wanted to ask you a couple questions:
- is your particular use case something that can be achieved through custom AMIs only, would the default Ubuntu AMI satisfy it?
- provided in the past you used to be able to deploy EKS managed nodes with custom Ubuntu AMI, have you ever faced any bugs / limitations in features compared to AL2 images?
@shivkumr , @Sathyam-Unicommerce - waiting for some feedback on the above.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
joining to this thread. answering your question - my particular use case can be achieved through custom AMIs only...
I have the eksctl version 0.161.0
and the problem is still persisting.
Error: could not create cluster provider from options: managedNodeGroups[0].overrideBootstrapCommand is required
when using a custom AMI based on Ubuntu2004 (managedNodeGroups[0].ami)
Why was this closed and with what resolution ? Also , I needed to add amiFamily too.
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: EKS-SANDBOX-0
region: eu-west-1
version: "1.28"
privateCluster:
enabled: true
additionalEndpointServices:
- "autoscaling"
- "logs"
vpc:
id: vpc123
securityGroup: sg-123
sharedNodeSecurityGroup: sg-123
manageSharedNodeSecurityGroupRules: true
subnets:
private:
eu-west-1a:
id: subnet-123
eu-west-1b:
id: subnet-123
eu-west-1c:
id: subnet-123
iam:
withOIDC: true
managedNodeGroups:
- name: EKS-NG-0
instanceType: t3.medium
# ami: ami-0a9b2251badf0cf47
# securityGroups:
# attachIDs: ["sg-1234"]
minSize: 1
desiredCapacity: 3
maxSize: 5
# amiFamily: AmazonLinux2
amiFamily: Ubuntu2004
ami: ami-008b919c7799c96e5
tags:
nodegroup-type: Ubuntu2004
maxPodsPerNode: 110
privateNetworking: true
ssh:
allow: false
volumeSize: 100
volumeName: /dev/xvda
volumeEncrypted: true
disableIMDSv1: true
taints:
- key: "node.cilium.io/agent-not-ready"
value: "true"
effect: "NoSchedule"
cloudWatch:
clusterLogging:
enableTypes: ["all"]
logRetentionInDays: 7
I have the eksctl version
0.161.0
and the problem is still persisting.Error: could not create cluster provider from options: managedNodeGroups[0].overrideBootstrapCommand is required when using a custom AMI based on Ubuntu2004 (managedNodeGroups[0].ami)
Why was this closed and with what resolution ? Also , I needed to add amiFamily too.
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: EKS-SANDBOX-0 region: eu-west-1 version: "1.28" privateCluster: enabled: true additionalEndpointServices: - "autoscaling" - "logs" vpc: id: vpc123 securityGroup: sg-123 sharedNodeSecurityGroup: sg-123 manageSharedNodeSecurityGroupRules: true subnets: private: eu-west-1a: id: subnet-123 eu-west-1b: id: subnet-123 eu-west-1c: id: subnet-123 iam: withOIDC: true managedNodeGroups: - name: EKS-NG-0 instanceType: t3.medium # ami: ami-0a9b2251badf0cf47 # securityGroups: # attachIDs: ["sg-1234"] minSize: 1 desiredCapacity: 3 maxSize: 5 # amiFamily: AmazonLinux2 amiFamily: Ubuntu2004 ami: ami-008b919c7799c96e5 tags: nodegroup-type: Ubuntu2004 maxPodsPerNode: 110 privateNetworking: true ssh: allow: false volumeSize: 100 volumeName: /dev/xvda volumeEncrypted: true disableIMDSv1: true taints: - key: "node.cilium.io/agent-not-ready" value: "true" effect: "NoSchedule" cloudWatch: clusterLogging: enableTypes: ["all"] logRetentionInDays: 7
Your issue is different than the issue that was closed. Your issue is being caused because when you specify an AMI, you must also include the overrideBootstrapCommand option in your yaml file.
See the documentation here: https://eksctl.io/usage/custom-ami-support/ And announcement for this requirement: https://eksctl.io/announcements/nodegroup-override-announcement/
$ eksctl create cluster -f ./cluster.yaml Error: when using a custom AMI, amiFamily needs to be explicitly set via config file or via --node-ami-family flag
Trying to create a eks cluster using the ubuntu custom AMI from canonical, eksctl is complaining to set amiFamily, but when I add amiFamily then the request fails with the below error.