[EKS] [request]: Allow feature gates to be set on master components

[jammerful]

Tell us about your request I would like to be able to set feature gates on master components such as the kube-apiserver, this would allow me to enable features early.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Right now I can't enable feature gates on any master components, while I have a particular feature gate in mind I would like the general ability to toggle feature flags upon creating/updating a cluster. I understand that the feature gates enable alpha features, so I would simply note in the documentation and a warning upon any feature flag being set that using this feature can make your cluster unstable and all feature gates set should be throughly tested before releasing to production.

The particular feature gate is the VolumeSnapshotDataSource which is needed for the EBS CSI Driver's snapshotting feature.

Are you currently working around this issue? I will deploy my own container to do the snapshotting myself.

[lanefu]

@jammerful hey I'm working on similar issue.... would you be able to explain your workaround a little more in-depth? What container are you deploying?

[jammerful]

@lanefu I'm assuming you want the volume snapshot feature, if you're trying to enable a random feature gate I don't think you can do that. For volume snapshots, you can use the now deprecated external storage code located here https://github.com/kubernetes-incubator/external-storage/tree/master/snapshot. I've been using it for the past few weeks, and it's working well. In the repo you will find the manifest to deploy and instructions on how to use the CRDs. Lastly, snapshotting in the CSI is beta in 1.17, so maybe by next spring/summer will be available on EKS.

[rinrailin]

Another good feature gate to be able to enable is VolumeSubpathEnvExpansion which is in GA for k8s v1.17 but in alpha for k8s v1.14

[dgarbus]

Shouldn't the label be EKS rather than ECS?

[jammerful]

@ellenthsu Please remove the ECS label and add the EKS label.

[jammerful]

Apparently alpha feature gates are not enabled on the stable channels of AKS or GKE, but they do have unstable channels that enable beta feature gates. Given that I don't know if AWS would consider doing this for production clusters, wish they would speak up.

[ellenthsu]

@ellenthsu Please remove the ECS label and add the EKS label.

done

[chris-vest]

I'm wanting to enable the ProcMountType feature gate in order to run Docker rootless, without having to use privileged pods.

[tokiwong]

I would like this feature to enable ServerSideApply

[spanky-medal]

We're looking for ServiceNodeExclusion since at scale, we can hit an AWS NLB service limit of 500 targets. The services we scale will not be used via the NLB (they're queue-based) so they just fill up NLB slots for no reason.

[themish95]

My cx is looking to enable SCTP and it requires the SCTPSupport feature gate

https://kubernetes.io/docs/concepts/services-networking/network-policies/#sctp-support

[hymgg]

Need to support SCTP for our telco apps too.

[mikeaorlando]

I am interested in this to enable Service Topology in EKS 1.17. I am following an example to reduce cross availability zone traffic (https://v1-17.docs.kubernetes.io/docs/concepts/services-networking/service-topology/#prefer-node-local-zonal-then-regional-endpoints) which require feature-flags to be accessible: https://v1-17.docs.kubernetes.io/docs/tasks/administer-cluster/enabling-service-topology/#enable-service-topology

[lanefu]

@mikeaorlando something that I discovered.....

If you're using terraform modules, it's easy to switch from EKS managed nodegroups to "worker groups". the level of config effort is about the same in terraform whether managed or unmanaged, and you can easily pass arguments for feature flags.. EX:

workers_group_defaults = {
  instance_type = "t3a.xlarge"
  key_name = "secret_key"
  kubelet_extra_args = "--feature-gates=ExpandCSIVolumes=true,ExpandPersistentVolumes=true"
}

[DanOfir]

@mikeaorlando @lanefu i am also trying to use Service Topology running on EKS 1.17, i created the feature gates as @lanefu described, how can i make sure the feature is enabled? @mikeaorlando did you succeeded using the feature in EKS at all? cause every time i create/edit service with the TopologyKeys it seems to be deleted after saving. any advice guys?

[chasmosis]

@mikeaorlando @DanOfir I am in the same boat as you wanting to use Service Topology on EKS 1.17 and find that since it has to be enabled at the API server label I'm unable to set that feature gate. We are wanting to use https://kubernetes.io/docs/concepts/services-networking/service-topology/#only-node-local-endpoints for host binding a service on only the nodes running the pods for that service.

[mikeaorlando]

I have been unable to use Service Topology running on EKS 1.17. My current workaround is to have all nodes in the cluster in one availability zone. This does not constrict traffic to be node-local, but reduces ingress and egress charges from cross-az traffic. My current plan is to wait until EKS 1.18.

This was helpful in determining which feature flags were set in the api-server: https://docs.aws.amazon.com/eks/latest/userguide/api-server-flags.html

[3oris]

In our case, we would want to set --feature-gates=IAMIdentityMappingCRD

[nicolai86]

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

[mbelang]

Would it be possible to have a generic way to enable any feature-gates in EKS?

I'm on 1.18 and want StartupProbes

[llamahunter]

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

Wait, AWS put out release notes specifically saying that pod topologySpreadConstraints were supported, but did not enable the feature gate? https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Pod Topology Spread has reached beta status. You can use topology spread constraints to control how pods are spread across your cluster among failure-domains such as Regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. For more information, see Pod Topology Spread Constraints in the Kubernetes documentation.

[nicolai86]

@llamahunter it seems beta features are enabled by default in k8s, so the feature gate doesn't need to be passed explicitly anymore.

[llamahunter]

@nicolai86 Have you successfully set pod topologySpreadConstraints using topology.kubernetes.io/zone topologyKeys in EKS v1.18?

[felixep]

Base on the documentation the current version supported is v1.18.9 and there is this bug that doesn't let you to get the results you are expecting from the topologySpreadConstraints. The fix was pushed to the v1.18.11 as per the release notes https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md, so there is nothing to do but wait until Amazon release support for v1.18.11.

[gabegorelick]

Wait, AWS put out release notes specifically saying that pod topologySpreadConstraints were supported, but did not enable the feature gate?

FYI, pod topology spread was removed from EKS docs in https://github.com/awsdocs/amazon-eks-user-guide/commit/fc59dad48cdc2258aa2070709218fde4a38d62bd. So the docs are technically accurate now (they don't claim it's supported).

[ankitjain28may]

Is there any way to disable feature gates in EKS, I want to disable server side apply feature gate.

[nonamef]

On 1.19 and would be good to have EphemeralContainers.

[themish95]

Is there any way to disable feature gates in EKS, I want to disable server side apply feature gate.

If you have a support plan, I think you can request them to

[themish95]

Definitely would like to see EphemeralContainersin v1.20 or the next release v1.21

[bitmexgmarkey]

I would appreciate having GenericEphemeralVolume made available.

[PatTheSilent]

Having TTLAfterFinished would be great. We use a lot of Jobs for database migrations and other administration tasks and they end up taking IPs, which for vpc-cni is quite a problem.

[gsaraf]

Interested in SizeMemoryBackedVolumes, currently alpha. Would be great to have a way to specify any feature flags.

Thanks!

[vinayus]

Would love to have MixedProtocolLBService enabled, helps us in having a single LB for service

[Noksa]

In a little more than a week we will be celebrating two years of this ticket. 👌

[bseenu]

Would be interested in the feature gate

HPAContainerMetrics

[pcgeek86]

I need to enable the ephemeral containers feature, so that I can use kubectl debug to debug a Pod. I can't enable this on Amazon EKS, so I will have to use a different clustering solution.

[ringerc]

Same here - but the same issue exists on Azure Kubernetes Service too:

• https://github.com/Azure/AKS/issues/1887
• https://github.com/Azure/AKS/issues/2058

It's more than a tad frustrating. It'd be understandable to limit support when alpha features are enabled, flag clusters "tainted" or something. But just not exposing it - or documenting the feature gates that are enabled, for that matter - is frustrating.

[cliffpracht]

We're also looking to experiment with the HPAContainerMetrics feature, which we currently can not enable on EKS.

[mariojacobo]

Would be interested in SupportPodPidsLimit feature gate

[behnm]

How can I enable feature gate on eks to use Topology Aware Hints ?

I enabled it on my custom-scheduler, But I think I should enableTopologyAwareHints=true on apiserver too.

[mohitreddy1996]

We are looking for HPAContainerMetrics. Enabling this would really help us with setting up HPA on our large EKS clusters

[alexppg]

Please, just add reactions to comments and/or subscribe, it's clear that we all want this, comment about it it's just annoying for all the subscribed people.

[amalendur]

I would appreciate having "BoundServiceAccountTokenVolume" made configurable on demand - either enable or disable.

[adabuleanu]

I would appreciate having "PodSecurity" made configurable on demand

[jcogilvie]

We'd like the ability to configure grpc health checks and API server tracing, if possible.

[Shivam0609]

After going through above comments, I believe still we haven't got a solution for enabling alpha features in eks master node.

[yuvalshi0]

GRPCContainerProbe seems to not be enabled on 1.24

[iandelahorne]

We would like to be able to enable MatchLabelKeysInPodTopologySpread when EKS gains support for 1.25

[m00lecule]

We are interested in MinDomainsInPodTopologySpread in EKS 1.25

[brian-provenzano]

i think this is more important now that EKS is disabling beta feature gates in 1.24+: https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-kubernetes-version-1-24/

We should be able to selectively enable beta features if desired in the control plane.