aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.21k stars 317 forks source link

[EKS] [request]: Increase/Customize reserved disk space on nodes #1847

Open MartinEmrich opened 1 year ago

MartinEmrich commented 1 year ago

Community Note

Tell us about your request Currently, the AWS managed nodegroup node images only reserve 1GB of local disk space for the OS/kubelet (configured in /etc/kubernetes/kubelet/kubelet-config.json). But the local operating system installation uses more than 2 GB. So even if ephemeral-storage resource requests/limits are configured adequately, a fully utilized node runs into Disk Pressure state.

AWS support asked me (Ticket 10768345041) to report the issue here.

Which service(s) is this request for? EKS (managed nodegroup node image)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Managed Nodes cannot be customized easily to override that value (and IMHO it should be set to a good value from the start)

Are you currently working around this issue? I configure slightly more ephemeral-storage resources that are actually needed on my workloads, but that increases waste and is not bullet-proof.

MartinEmrich commented 1 year ago

Indeed the "workaround" provided by AWS support today was to override the config with --kube-reserved=ephemeral-storage=3Gi --system-reserved=ephemeral-storage=3Gi in kubelet-extra-args. As eksctl does not provide a sane way to override this, I cannot implement that workaround.