search

axboe / fio

Flexible I/O Tester
GNU General Public License v2.0
5.23k stars 1.26k forks source link

nrfiles=2 in rw=randwrite writes only in single file #1676

Open Tulsishah opened 10 months ago

Tulsishah commented 10 months ago 
[global]
ioengine=libaio
direct=1
fadvise_hint=0
verify=0
rw=read
bs=1M
iodepth=64
invalidate=1
ramp_time=10s
runtime=60s
startdelay=2m
time_based=1
nrfiles=1
thread=1
fsync=1
openfiles=1
group_reporting=1
allrandrepeat=1
filename_format=$jobname.$jobnum.$filenum

[3mb_rand_write]
stonewall
startdelay=310
directory=gcs/3mb
filesize=3M
rw=randwrite
nrfiles=2
fsync=1
numjobs=1

writes 3mb on single file(not 2 files) in case of random writes.

May I know the reason for the same?

vincentkfu commented 10 months ago

This appears to be a bug in fio. Probably this is related to the other issue you filed as well.

vincent@fio-dev:~/fio-dev/1676$ ./fio-canonical/fio 1676.fio --debug=io &> 1676.log
vincent@fio-dev:~/fio-dev/1676$ head 1676.log -n 100
fio: set debug option io
io       4466  load ioengine libaio
3mb_rand_write: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.36-19-gbdf99
Starting 1 thread
3mb_rand_write: Laying out IO files (2 files / total 6MiB)
io       4466  declare unneeded cache 3mb_rand_write.0.0: 0/3145728
io       4466  declare unneeded cache 3mb_rand_write.0.1: 0/3145728
io       4468  declare unneeded cache 3mb_rand_write.0.0: 0/3145728
io       4468  fill: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x200000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x200000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x0,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x0,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x0,ddir=3,file=3mb_rand_write.0.0
io       4468  io_u 0x5580635816c0, setting file failed
io       4468  get_io_u failed
io       4468  fill: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  prep: io_u 0x5580635816c0: ret=0
io       4468  queue: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
io       4468  calling ->commit(), depth 1
io       4468  io_u 0x558063581500, setting file failed
io       4468  get_io_u failed
io       4468  io_u_queued_complete: min=1
io       4468  getevents: 1
io       4468  complete: io_u 0x5580635816c0: off=0x100000,len=0x100000,ddir=1,file=3mb_rand_write.0.0
Tulsishah commented 10 months ago

When can we expect a fix for this issue?

vincentkfu commented 10 months ago

There is no timeline.

To do a good job of fixing this the steps would be:

  1. Establish a set of requirements for fio's behavior involving multiple files and the options affecting this.
  2. Write a test script to make sure fio meets these requirements.
  3. Write a patch to fix this issue that causes no regressions.

One thing that might help is if someone tried older fio versions to see if this is a longstanding or a recent problem. If this is a regression, then it might help to identify the patch introducing this issue. It would also help to have a simpler job file that can be used to produce this issue (https://stackoverflow.com/help/minimal-reproducible-example).

Tulsishah commented 10 months ago

I tested fio versions 3.34 and 3.35, and both exhibited the reported issues.(https://github.com/axboe/fio/issues/1676, https://github.com/axboe/fio/issues/1677)

Tulsishah commented 4 months ago

This issue also occurs in rw=write mode. However, using timebased=0 it works as expected.