Improve performance of block device

[heiher]

Allow block requests that single vector to parallel execution.

Benchmark results fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Before: fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=30.0MiB/s,w=10.1MiB/s][r=7924,w=2580 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=295: Fri Dec 14 15:47:21 2018 read: IOPS=7800, BW=30.5MiB/s (31.0MB/s)(3070MiB/100750msec) bw ( KiB/s): min=23656, max=32712, per=99.95%, avg=31186.70, stdev=1449.67, samples=201 iops : min= 5914, max= 8178, avg=7796.65, stdev=362.45, samples=201 write: IOPS=2607, BW=10.2MiB/s (10.7MB/s)(1026MiB/100750msec); 0 zone resets bw ( KiB/s): min= 7936, max=11832, per=99.96%, avg=10423.31, stdev=570.12, samples=201 iops : min= 1984, max= 2958, avg=2605.80, stdev=142.53, samples=201 cpu : usr=2.29%, sys=23.94%, ctx=757029, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs): READ: bw=30.5MiB/s (31.0MB/s), 30.5MiB/s-30.5MiB/s (31.0MB/s-31.0MB/s), io=3070MiB (3219MB), run=100750-100750msec WRITE: bw=10.2MiB/s (10.7MB/s), 10.2MiB/s-10.2MiB/s (10.7MB/s-10.7MB/s), io=1026MiB (1076MB), run=100750-100750msec

Disk stats (read/write): vda: ios=784800/262572, merge=26/27, ticks=4808917/1593619, in_queue=6402910, util=99.97%

After: fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=64.9MiB/s,w=21.1MiB/s][r=16.6k,w=5397 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=313: Fri Dec 14 15:49:39 2018 read: IOPS=16.2k, BW=63.1MiB/s (66.2MB/s)(3070MiB/48623msec) bw ( KiB/s): min=57512, max=75872, per=99.96%, avg=64630.28, stdev=2634.57, samples=97 iops : min=14378, max=18968, avg=16157.55, stdev=658.64, samples=97 write: IOPS=5401, BW=21.1MiB/s (22.1MB/s)(1026MiB/48623msec); 0 zone resets bw ( KiB/s): min=18384, max=25400, per=99.97%, avg=21600.74, stdev=936.80, samples=97 iops : min= 4596, max= 6350, avg=5400.14, stdev=234.20, samples=97 cpu : usr=6.61%, sys=77.06%, ctx=1336, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs): READ: bw=63.1MiB/s (66.2MB/s), 63.1MiB/s-63.1MiB/s (66.2MB/s-66.2MB/s), io=3070MiB (3219MB), run=48623-48623msec WRITE: bw=21.1MiB/s (22.1MB/s), 21.1MiB/s-21.1MiB/s (22.1MB/s-22.1MB/s), io=1026MiB (1076MB), run=48623-48623msec

Disk stats (read/write): vda: ios=781846/261555, merge=0/18, ticks=286748/79838, in_queue=393567, util=99.84%

[dinamic]

Wow, that's good 100% I/O improvement! 🥇

I think it would have big impact on the performance of software that relies on xhyve - take docker for macOS as an example. Can't wait to have it merged!

@jeremyhu I can't hold myself but to ping you. Sorry for the noise :)

[dinamic]

@heiher I'm trying to understand why this PR got closed. Has it been implemented and merged outside of this PR?

[heiher]

As the maintainers have not replied for a long time, I believe that there may be a better way available, so I have closed this PR.