search

eksctl-io / eksctl

The official CLI for Amazon EKS
https://eksctl.io
Other
4.92k stars 1.41k forks source link

Trying to verify --enable-cadvisor-json-endpoints is passed on a 1.18 built cluster #3329

Closed hgrant-ebsco closed 3 years ago

hgrant-ebsco commented 3 years ago

We are deploying EKS clusters with v1.18, which has the cadvisor json endpoint disabled by default. On our clusters, we have one tool that needs this endpoint for the moment, so to enable this, I am trying to use the --enable-cadvisor-json-endpoints kubelet flag.

What help do you need?

The nodegroups build successfully, and are functional. Before I deploy the tool needing the flag, I want to verify the flag was actually applied, but I cannot see any reference to it in my kubelet config on any of the nodes. My Eksctl nodegroup config is as follows for the kubelet extra args:

    kubeletExtraConfig:
      kubeReserved:
        cpu: "500m"
        memory: "500Mi"
        ephemeral-storage: "1Gi"
      kubeReservedCgroup: "/kube-reserved"
      systemReserved:
        cpu: "500m"
        memory: "500Mi"
        ephemeral-storage: "1Gi"
      evictionHard:
        memory.available:  "600Mi"
        nodefs.available: "10%"
      enable-cadvisor-json-endpoints: true

The error thrown is: [✖] validating generated KubeletConfiguration object: error unmarshaling JSON: while decoding JSON: json: unknown field "enable-cadvisor-json-endpoints"

Callisto13 commented 3 years ago

Hi @JayG-EBSCO, thanks for opening this issue.

~I believe that flag would qualify as a feature gate, could you try adding it like this instead?~

kubeletExtraConfig:
    ...
    featureGates:
        EnableCAdvisorJsonEndpoints: true

Edit: scratch that, does not work. I will keep digging.

Callisto13 commented 3 years ago

So from what I understand, the kubelet config does not expose that flag as a field, and it does not count as a feature gate. This could be because that flag is marked for removal in v1.21.

Eksctl sets up the kubelet to start from a config file, we do have some other flags around bootstrapping, but it is unlikely that we will add direct support for a deprecated flag. One option may be to use overrideBootstrapCommand which I will test now.

hgrant-ebsco commented 3 years ago

So from what I understand, the kubelet config does not expose that flag as a field, and it does not count as a feature gate. This could be because that flag is marked for removal in v1.21.

Eksctl sets up the kubelet to start from a config file, we do have some other flags around bootstrapping, but it is unlikely that we will add direct support for a deprecated flag. One option may be to use overrideBootstrapCommand which I will test now.

We have discovered that the 3rd party has released a beta version that negates the need for the flag, and have tested it successfully. This is no longer an issue.

Callisto13 commented 3 years ago

Ha! Excellent 🎉