[Feature] NVMe configuration for L8s machines

chris-milsted commented 1 year ago

I want to use machines with local NVMe drives for solutions like https://cloudnative-pg.io/ on AKS clusters. I want to be able to use a CSI provider which offers pools so I can select local NVMe disks or Azure disks. As an example Ondat

There are some example for other providers where "preBootstrap" commands can be run to do things like:

    preBootstrapCommands:
      - mkdir -p /var/lib/storageos
      - echo "/dev/nvme1n1 /var/lib/storageos ext4 defaults,discard 0 1" >> /etc/fstab
      - mkfs.ext4 /dev/nvme1n1
      - mount /var/lib/storageos

This would be run on the nodes as part of the post node creation and pre K8s join steps I assume.

Currently experimenting with kubectl debug does not work (do not have system admin access to work with block devices). See discussion in ISSUE 1541.

There are times when local NVMe storage and the lower latency would be beneficial for databases.

Thanks

andyzhangx commented 1 year ago

Does this open source solution work for you? https://github.com/Azure/kubernetes-volume-drivers/tree/master/local

chris-milsted commented 1 year ago

Hi,

Unfortunately not, we want a bit more control of the NVMe drive and to be able to slice it up into different PVs for different parts of the database. e.g. the Write Ahead Logs and Data stores from a number of databases would all be able to share a single NVMe using lots of PVs given the specs of the NVMe drives. This means we want to map a single NVMe drive on a single K8s node to many pods/containers in the cluster. We also want to be able to run our own CSI on this NVMe drive as then we can do things like replicate the data from an NVMe drive in one machine to another machine for resilience.

Ideally we want to have this operate at the Linux node level, i.e. we are looking for a solution which would allow Linux nodes to have customisations and then a CSI solution could then be applied using a daemon set.

The solution that exists as I read it can only map a 1:1 disk or partition to a pod? Also, currently there does not seem to be any way with kubectl debug to interact with the devices so this is really just mapping a single disk to a single pod. (e.g. fdisk /dev/nvme0n1 does not work with the kubectl debug containers).

Chris