search

spdk / spdk

Storage Performance Development Kit
https://spdk.io/
Other
3.04k stars 1.19k forks source link

SPDK vhost only uses a single core to service a VM #2814

Open xinyangge-db opened 1 year ago

xinyangge-db commented 1 year ago

Sighting report

Our setup uses the SPDK vhost to create a virtio-blk device for a VM with multi-queue enabled. The backend of the virtio-blk device is an AIO bdev. Below is a screenshot taken from the guest.

image

When we benchmark the disk performance from the VM using FIO, we noticed that the spdk_tgt process only services the VM's IO requests using a single core while we have dedicated three cores to it. This limits the IOPS of that VM to whatever a single core can achieve on the host using aio. Below are the screenshots that show only one SPDK core (cpu1) having intensive kernel executions for AIO (the upper half is in the guest and the bottom half runs top from the host).

num_jobs=1 image

num_jobs=2 image

Expected Behavior

We expect all SPDK cores to be able to process the I/O requests from the same VM (especially when the CPU is becoming a bottleneck). We are not sure if this is by design or due to misconfigurations from our side.

Current Behavior

Possible Solution

Steps to Reproduce

  1. This is how we launch SPDK: /spdk_deploy/build/bin/spdk_tgt -S /var/run/kata-containers/vhost-user/block/sockets -m [1,2,3] --huge-dir /databricks/csidata/spdk_hugetlbfs --iova-mode va -L all
  2. We didn't specify --cpumask when running rpc.py vhost_create_blk_controller

Context (Environment including OS version, SPDK version, etc.)

changpe1 commented 1 year ago

@xinyangge-db it's not a bug but SPDK's intention to do like this, but as discussed, someone is working to add the core scalability to vhost-blk target. Here is the patch draft link https://review.spdk.io/gerrit/c/spdk/spdk/+/14196

xinyangge-db commented 1 year ago

@changpe1 This is encouraging to see. We look forward to using this feature!

yewhen commented 1 year ago

@xinyangge-db it's not a bug but SPDK's intention to do like this, but as discussed, someone is working to add the core scalability to vhost-blk target. Here is the patch draft link https://review.spdk.io/gerrit/c/spdk/spdk/+/14196

Will SPDK consider apply this patch set in the future?

changpe1 commented 1 year ago

@xinyangge-db it's not a bug but SPDK's intention to do like this, but as discussed, someone is working to add the core scalability to vhost-blk target. Here is the patch draft link https://review.spdk.io/gerrit/c/spdk/spdk/+/14196

Will SPDK consider apply this patch set in the future?

Yes, this feature is only for very limited scenarios, we will provide the parameter to users.

cheese commented 1 year ago

@xinyangge-db it's not a bug but SPDK's intention to do like this, but as discussed, someone is working to add the core scalability to vhost-blk target. Here is the patch draft link https://review.spdk.io/gerrit/c/spdk/spdk/+/14196

Any news? This PR was abandoned.

cheese commented 1 year ago

I found the active draft https://review.spdk.io/gerrit/c/spdk/spdk/+/16068 and had a quit test which showed excellent.