vincentkfu
commented
2 years ago Do md devices support polling (hipri)?
Closed qiaobaaa closed 2 years ago
vincentkfu
commented
2 years ago Do md devices support polling (hipri)?
axboe
commented
2 years ago Do md devices support polling (hipri)?
They do in later kernels, but not in 5.11.
qiaobaaa
commented
2 years ago Which specific version of the latest Linux kernel refers to? Is 5.17.10 ok?
qiaobaaa
commented
2 years ago Do md devices support polling (hipri)?
They do in later kernels, but not in 5.11.
At present, I want to install the kernel 5.17.10 and test it again
Please acknowledge the following before creating a ticket
Description of the bug: When I use the io_uring engine to test the raid5 volume with the fllowing script (use polled IO completions), it will report an error, but if I not use polled IO completions , will not report an error . Md5 is a raid5 array, which consists of three working disks nvme0n1, nvme1n1 and nvme2n1 and a free nvme3n1.
4k-randread-16-1.fio script (use polled IO completions): [global] direct=1 ioengine=io_uring size=200G runtime=1m time_based=1 ramp_time=1s norandommap randrepeat=0 stonewall group_reporting=1 per_job_logs=0 filename=/dev/md5 log_avg_msec=500 hipri=1
[job 1] name=RR_lat bs=4k numjobs=1 iodepth=16 rw=randread write_lat_log=RR
Test md5: sivp@sivp-NF5180M6:~/fio-test-raid5/io_uring_io/4k_read/16depeth$ sudo fio 4k-randread-16-1.fio RR_lat: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=16 fio-3.29-146-gc822 Starting 1 process fio: io_u error on file /dev/md5: Operation not supported: read offset=67879084032, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=105565327360, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=160620691456, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=67406090240, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=159223758848, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=157909307392, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=410456064, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=30777634816, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=168153309184, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=62829453312, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=186625892352, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=24743829504, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=133804670976, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=207737004032, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=172864450560, buflen=4096 fio: io_u error on file /dev/md5: Operation not supported: read offset=176234332160, buflen=4096 fio: pid=26595, err=95/file:io_u.c:1846, func=io_u error, error=Operation not supported
RR_lat: (groupid=0, jobs=1): err=95 (file:io_u.c:1846, func=io_u error, error=Operation not supported): pid=26595: Fri May 27 03:11:31 2022 cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=18 IO depths : 1=6.2%, 2=12.5%, 4=25.0%, 8=50.0%, 16=6.2%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=33.3%, 4=33.3%, 8=0.0%, 16=33.3%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=16,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
Disk stats (read/write): md5: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% nvme0n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% nvme3n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% nvme2n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
Not use polled IO completions: 4k-randread-16-1.fio script : [global] direct=1 ioengine=io_uring size=200G runtime=1m time_based=1 ramp_time=1s norandommap randrepeat=0 stonewall group_reporting=1 per_job_logs=0 filename=/dev/md5 log_avg_msec=500
[job 1] name=RR_lat bs=4k numjobs=1 iodepth=16 rw=randread write_lat_log=RR
results: sivp@sivp-NF5180M6:~/fio-test-raid5/io_uring_io/4k_read/16depeth$ sudo fio 4k-randread-16-1.fio RR_lat: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=16 fio-3.29-146-gc822 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=713MiB/s][r=183k IOPS][eta 00m:00s] RR_lat: (groupid=0, jobs=1): err= 0: pid=26317: Fri May 27 03:10:33 2022 read: IOPS=183k, BW=714MiB/s (749MB/s)(41.9GiB/60001msec) slat (nsec): min=415, max=130549, avg=1324.31, stdev=923.18 clat (usec): min=12, max=997, avg=85.57, stdev=15.26 lat (usec): min=13, max=998, avg=87.01, stdev=15.26 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 77], | 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 92], 80.00th=[ 96], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 135], 99.50th=[ 145], 99.90th=[ 165], 99.95th=[ 174], | 99.99th=[ 212] bw ( KiB/s): min=700280, max=753592, per=100.00%, avg=731928.85, stdev=9184.24, samples=119 iops : min=175070, max=188398, avg=182982.04, stdev=2295.93, samples=119 lat (usec) : 20=0.01%, 50=0.03%, 100=87.22%, 250=12.74%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% cpu : usr=23.29%, sys=42.82%, ctx=3336426, majf=0, minf=70 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10971600,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs): READ: bw=714MiB/s (749MB/s), 714MiB/s-714MiB/s (749MB/s-749MB/s), io=41.9GiB (44.9GB), run=60001-60001msec
Disk stats (read/write): md5: ios=11155895/0, merge=0/0, ticks=814344/0, in_queue=814344, util=99.91%, aggrios=2788973/0, aggrmerge=0/0, aggrticks=208767/0, aggrin_queue=208767, aggrutil=99.89% nvme0n1: ios=3719450/0, merge=0/0, ticks=291474/0, in_queue=291475, util=99.88% nvme3n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% nvme2n1: ios=3718164/0, merge=0/0, ticks=246785/0, in_queue=246784, util=99.88% nvme1n1: ios=3718281/0, merge=0/0, ticks=296812/0, in_queue=296812, util=99.89%
Environment: Linux sivp-NF5180M6 5.11.0-051100-generic #202102142330 SMP Sun Feb 14 23:33:21 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
fio version: fio-3.29-146-gc822
Reproduction steps sudo fio 4k-randread-16-1.fio