search

clincha-org / clincha

Configuration and monitoring of clinch-home infrastructure
https://clinch-home.com
1 stars 1 forks source link

Storage testing #113

Closed clincha closed 11 months ago

clincha commented 1 year ago

I'm hitting issues that I believe are caused by high disk latency. This needs to be resolved so that the Kubernetes cluster is performing well for the applications that sit on it.

clincha commented 1 year ago

This looks useful for testing out the disks: https://access.redhat.com/solutions/4885641

clincha commented 1 year ago

Ceph HDD

image

Ceph SSD

image

Local NVME

image
clincha commented 1 year ago

After setting the MTU to 9000 I'm seeing inconsistent results with the SSD pool.

First run

INFO: 99th percentile of fsync is 371195904 ns
WARN: 99th percentile of the fsync is greater than the recommended value which is 371195904 ns > 20 ms, faster disks are recommended to host etcd for better performance

Second run

INFO: 99th percentile of fsync is 299892736 ns
WARN: 99th percentile of the fsync is greater than the recommended value which is 299892736 ns > 20 ms, faster disks are recommended to host etcd for better performance

Third run

INFO: 99th percentile of fsync is 19267584 ns
INFO: 99th percentile of the fsync is within the recommended threshold: - 20 ms, the disk can be used to host etcd
clincha commented 11 months ago

Running on local-lvm for now. That space is easily expandable and quick but makes the guests sticky to a host. This is fine with Kubernetes managing applications