search

digitalocean / DOKS

Managed Kubernetes designed for simple and cost effective container orchestration.
https://www.digitalocean.com/products/kubernetes/
Apache License 2.0
80 stars 4 forks source link

Resizing root partition of a DOKS Worker Node Droplet #27

Open artem-zinnatullin opened 3 years ago

artem-zinnatullin commented 3 years ago

Hi, we're trying to configure DOKS for our disk I/O workloads. For this reason we use droplets with NVMe disks — Basic Premium Droplets.

The problem is that these Droplets boot with entire NVMe drive mounted as / root ext4 partition.

For example we use Basic Premium Droplet with 50GB NVMe and it's partitioned like this after boot:

fdisk -l
Disk /dev/vda: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3FAA642C-E0CB-4615-8382-0482DE0DF9B0

Device     Start       End   Sectors Size Type
/dev/vda1   6144 104857566 104851423  50G Linux filesystem
/dev/vda2   2048      6143      4096   2M BIOS boot

Partition table entries are not in disk order.

For proper use of local drives in k8s with CSI like https://github.com/minio/direct-csi we need to have separate partitions for data:

like so:

fdisk -l
Disk /dev/vda: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3FAA642C-E0CB-4615-8382-0482DE0DF9B0

Device        Start       End  Sectors Size Type
/dev/vda1      6144  31463423 31457280  15G Linux filesystem
/dev/vda2      2048      6143     4096   2M BIOS boot
/dev/vda3  31463424 104857566 73394143  35G Linux filesystem

Where 35GB drive /dev/vda3 and its partition are used solely for data storage and we can provide it to direct-csi or other k8s storage solution as a separate drive.

However we can not reduce or shrink size of the root / ext4 partition, it requires boot in safe mode otherwise root partition becomes read-only after resize on live system.

I've tried resizing / with fdisk but then resize2fs gives error:

resize2fs /dev/vda1
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/vda1 is mounted on /; on-line resizing required

Basically ext4 partition needs to be unmounted for it to be reduced, but unmounting root / partition safely is nearly impossible, we'd need to stop all software that can write to disk, then create tmpfs with root data, unmount root, resize it, mount back, move tmp data back, very unsafe.

Ideally we'd be able to do that with either custom ISO or node initialization script or DOKS could allow us configure partitions upfront when creating a worker Node so that it loads with correct partitions right away.

Related issue: https://github.com/digitalocean/csi-digitalocean/issues/384

Can you please suggest any workarounds? Thanks!