ceph-csi very slow on vm

[plano-fwinkler]

proxmox with ceph and talos as vm with ceph csi is much slower than openebs-hostpath, are there any modules missing for the kernel?

Environment

• Talos version: 1.8.2
• Kubernetes version: 1.31.2
• Platform: proxmox with ceph storage

[smira]

The issue you posted doesn't have any relevant details, including the performance numbers, the way you set up things, etc.

Ceph is a complicated subject, and setting it up properly is not trivial.

[plano-fwinkler]

We have a Proxmox Cluster with 5 Nodes and a Ceph Cluster on the Proxmox. The Ceph Cluster has a 100GB nic.

if i testing with kubestr fio:

with a local path Storageclass ` ./kubestr fio -s openebs-hostpath PVC created kubestr-fio-pvc-qqb7w Pod created kubestr-fio-pod-4z7zc Running FIO test (default-fio) on StorageClass (openebs-hostpath) with a PVC of Size (100Gi) Elapsed time- 28.089900025s FIO test results:

FIO version - fio-3.36 Global options - ioengine=libaio verify=0 direct=1 gtod_reduce=1

JobName: read_iops blocksize=4K filesize=2G iodepth=64 rw=randread read: IOPS=49767.750000 BW(KiB/s)=199087 iops: min=41961 max=61272 avg=49501.585938 bw(KiB/s): min=167847 max=245088 avg=198006.484375

JobName: write_iops blocksize=4K filesize=2G iodepth=64 rw=randwrite write: IOPS=21245.320312 BW(KiB/s)=84993 iops: min=9028 max=39728 avg=35385.707031 bw(KiB/s): min=36112 max=158912 avg=141543.125000

JobName: read_bw blocksize=128K filesize=2G iodepth=64 rw=randread read: IOPS=36891.605469 BW(KiB/s)=4722663 iops: min=31849 max=45298 avg=36709.964844 bw(KiB/s): min=4076761 max=5798144 avg=4698881.500000

JobName: write_bw blocksize=128k filesize=2G iodepth=64 rw=randwrite write: IOPS=33320.179688 BW(KiB/s)=4265520 iops: min=17652 max=40996 avg=33119.656250 bw(KiB/s): min=2259456 max=5247488 avg=4239321.500000

Disk stats (read/write): sda: ios=1454972/1046364 merge=0/22 ticks=1907168/1466570 in_queue=3393654, util=29.229431%

• OK `

and with the ceph block Storageclass: rbd.csi.ceph.com

` ./kubestr fio -s ceph-block PVC created kubestr-fio-pvc-n7m9z Pod created kubestr-fio-pod-4jnqw Running FIO test (default-fio) on StorageClass (ceph-block) with a PVC of Size (100Gi) Elapsed time- 27.566283667s FIO test results:

FIO version - fio-3.36 Global options - ioengine=libaio verify=0 direct=1 gtod_reduce=1

JobName: read_iops blocksize=4K filesize=2G iodepth=64 rw=randread read: IOPS=242.109741 BW(KiB/s)=983 iops: min=98 max=496 avg=257.322571 bw(KiB/s): min=392 max=1987 avg=1030.129028

JobName: write_iops blocksize=4K filesize=2G iodepth=64 rw=randwrite write: IOPS=224.676819 BW(KiB/s)=914 iops: min=2 max=768 avg=264.464294 bw(KiB/s): min=8 max=3072 avg=1058.357178

JobName: read_bw blocksize=128K filesize=2G iodepth=64 rw=randread read: IOPS=213.964386 BW(KiB/s)=27884 iops: min=90 max=462 avg=223.967743 bw(KiB/s): min=11520 max=59254 avg=28694.708984

JobName: write_bw blocksize=128k filesize=2G iodepth=64 rw=randwrite write: IOPS=219.214661 BW(KiB/s)=28548 iops: min=4 max=704 avg=258.035706 bw(KiB/s): min=512 max=90112 avg=33048.785156

Disk stats (read/write): rbd2: ios=8696/8655 merge=0/267 ticks=2245425/1975831 in_queue=4221257, util=99.504547%

• OK

`

The talos machine has two nic's. One only to communicating with the ceph monitor's.

It's Working, but i think to slow.

[smira]

Then you need to dig further to understand why - what is the bottleneck, certainly Ceph block storage should be slower as it goes via the network, does replication, etc.

You can watch resource utilization to understand what is the bottleneck.

We are not aware of anything missing from the Talos side, and we do use Ceph a lot ourselves with Talos.