search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.83k stars 4.64k forks source link

Use local instance storage as root volume #15451

Open ddelange opened 1 year ago

ddelange commented 1 year ago

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Hi!👋

kOps can safely mount the local instance storage as root volume, instead of EBS, for a substantial performance boost whilst saving money.

2. Feel free to provide a design supporting your feature request.

Pasting the motivation and afaik the current state:

I have some containers that need fast temporary storage (around 100GB) we were using gp2 type AWS EBS volumes however they would quickly run out of burst balance. Local instance storage seemed like the perfect replacement as it would reduce the spend on slow EBS volumes and provide fast temporary storage. However I quickly found that Kubernetes doesn't seem to have quite implemented a way to use the local instance storage yet.

I wanted to use emptyDir volumes on my containers that needed fast local temporary storage so I tried moving /var/lib/kubelet to local instance storage by specifying it in the instance group configuration:

  volumeMounts:
    - device: /dev/nvme1n1
      filesystem: ext4
      path: /var/lib/kubelet

However like previous posters have mentioned I started seeing issues with disk pressure and the pods being evicted even though the local instance storage had only used 35% capacity.

Instead we have now switched to using a hostPath volume with an initContainer to set the correct permissions in the host directory. Our kops instance group now looks like this:

  volumeMounts:
    - device: /dev/nvme1n1
      filesystem: ext4
      path: /mnt/localssd

Relevant container configuration:

      initContainers:
      - name: fix-tmp-perms
        image: busybox
        securityContext:
          runAsUser: 0
        command: ["sh", "-c", "chown -R 201:201 /tmp/worker-temp; chmod 1777 /tmp/worker-temp; rm -rf /tmp/worker-temp/*"]
        volumeMounts:
        - name: worker-temp
          mountPath: /tmp/worker-temp

      volumes:
        - name: worker-temp
          hostPath:
            path: /mnt/localssd/worker-temp
            type: DirectoryOrCreate

What would be nice is to be able to specify the root volume in kops to use the local instance storage rather than having to be backed by EBS. I think this makes sense as the EBS volume is only used for temporary storage and is deleted when the instance is deleted.

Originally posted by @elliotdobson in https://github.com/kubernetes/kops/issues/429#issuecomment-689803817

ddelange commented 1 year ago

Re-reading the linked thread: I think this might me more of a request for documentation actually.

What is the kOps recommended way to opt in whenever a node from the (mixed) instance group happens to have local instance storage? E.g. for the IG "c6g.8xlarge", "c6gd.8xlarge", "c7g.8xlarge" which contains 2 regular nodes and 1 with NVMe.

And the same question goes for the control-plane nodes, although it's probably analogous :)

ddelange commented 1 year ago

This would potentially also allow swap (for which EBS is prohibitively slow): https://kubernetes.io/docs/concepts/architecture/nodes/#swap-memory

xref https://github.com/kubernetes/kops/issues/3251, https://github.com/kubernetes/kops/issues/14950

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ddelange commented 7 months ago

/remove-lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ddelange commented 4 months ago

/remove-lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

ddelange commented 1 month ago

/remove-lifecycle stale