search

ceph / ceph-nvmeof

Service to provide Ceph storage over NVMe-oF/TCP protocol
GNU Lesser General Public License v3.0
85 stars 46 forks source link

cpu affinity should be supported for the threads of io_context_pool and msgr-worker-* #38

Open liuqinfei opened 2 years ago

liuqinfei commented 2 years ago

Hi. I try to deploy the ceph-nvmeof and do some performance tests. I find that setting the cpu affinity for the threads of io_context_pool and msgr-worker-* can contribute to much higher performance both for the aarch64/x86 machines . Further more, for aarch64 machines, the performance result showes that deploying the reactor threads and the io_context_pool threads in the same ccl (core cluster, usually 4 cores/ccl) will get better performance that in different ccl. So, implementing this by default and give a parameter for choice in the ceph-nvmeof.conf may help users to get better performance.

PepperJo commented 2 years ago

Thanks for bringing this up. We already have an internal work item on this issue. We plan to support a CPU mask when creating a Ceph cluster context in SPDK.

epuertat commented 1 year ago

https://review.spdk.io/gerrit/c/spdk/spdk/+/16451

PepperJo commented 1 year ago

The patch has been merged in SPDK but work is required to expose it through the GW.