Stack trace when specifying higher iodepth values

[eksklawa]

I did run bench_fio command: ~/fio-plot/benchmark_script/bench_fio --target /tmp/testdir --type directory --mode randread randwrite --output . --iodepth 1 2 4 8 16 32 64 128 256 512 1024 2048 --numjobs 1 -s 20M -j ~/fio-plot/benchmark_script/fio-job-template.fio

which produced folder with logs and json files:

[root@centos1 tmp]# find . -name "*.json" -ls
 17679641      8 -rw-r--r--   1  root     root         7369 Sep  5 09:51 ./testdir/4k/randread-1-1.json
 17679679      8 -rw-r--r--   1  root     root         7375 Sep  5 09:51 ./testdir/4k/randread-2-1.json
 17696196      8 -rw-r--r--   1  root     root         7354 Sep  5 09:51 ./testdir/4k/randread-4-1.json
 17696202      8 -rw-r--r--   1  root     root         7355 Sep  5 09:51 ./testdir/4k/randread-8-1.json
 17696213      8 -rw-r--r--   1  root     root         7361 Sep  5 09:51 ./testdir/4k/randread-16-1.json
 17696219      8 -rw-r--r--   1  root     root         7371 Sep  5 09:51 ./testdir/4k/randread-32-1.json
 17696225      8 -rw-r--r--   1  root     root         7378 Sep  5 09:51 ./testdir/4k/randread-64-1.json
 17696231      8 -rw-r--r--   1  root     root         7376 Sep  5 09:51 ./testdir/4k/randread-128-1.json
 17696237      8 -rw-r--r--   1  root     root         7399 Sep  5 09:51 ./testdir/4k/randread-256-1.json
 17696243      8 -rw-r--r--   1  root     root         7408 Sep  5 09:51 ./testdir/4k/randread-512-1.json
 17696249      8 -rw-r--r--   1  root     root         7414 Sep  5 09:51 ./testdir/4k/randread-1024-1.json
 17696255      8 -rw-r--r--   1  root     root         7417 Sep  5 09:51 ./testdir/4k/randread-2048-1.json
 17696325      8 -rw-r--r--   1  root     root         7331 Sep  5 09:51 ./testdir/4k/randwrite-1-1.json
 17696331      8 -rw-r--r--   1  root     root         7340 Sep  5 09:51 ./testdir/4k/randwrite-2-1.json
 17696337      8 -rw-r--r--   1  root     root         7350 Sep  5 09:51 ./testdir/4k/randwrite-4-1.json
 17696343      8 -rw-r--r--   1  root     root         7356 Sep  5 09:51 ./testdir/4k/randwrite-8-1.json
 17696349      8 -rw-r--r--   1  root     root         7368 Sep  5 09:51 ./testdir/4k/randwrite-16-1.json
 17696355      8 -rw-r--r--   1  root     root         7376 Sep  5 09:51 ./testdir/4k/randwrite-32-1.json
 17696361      8 -rw-r--r--   1  root     root         7390 Sep  5 09:52 ./testdir/4k/randwrite-64-1.json
 17696367      8 -rw-r--r--   1  root     root         7407 Sep  5 09:52 ./testdir/4k/randwrite-128-1.json
 17696373      8 -rw-r--r--   1  root     root         7402 Sep  5 09:52 ./testdir/4k/randwrite-256-1.json
 17768760      8 -rw-r--r--   1  root     root         7416 Sep  5 09:52 ./testdir/4k/randwrite-512-1.json
 17768766      8 -rw-r--r--   1  root     root         7420 Sep  5 09:52 ./testdir/4k/randwrite-1024-1.json
 17763012      8 -rw-r--r--   1  root     root         7421 Sep  5 09:52 ./testdir/4k/randwrite-2048-1.json

However when I run fio_plot command:

 ~/fio-plot/fio_plot/fio_plot -i ./testdir/4k/ -T "TESTRUN" -d 1 2 4 8 16 32 64 128 256 512 1024 2048 -L -t iops -r randread

or ~/fio-plot/fio_plot/fio_plot -i ./testdir/4k/ -T "TESTRUN" -L -t iops -r randread

it produces only graph for iodepths 1-64

I found it fails on iodepth higher than 64

[root@centos1 tmp]# ~/fio-plot/fio_plot/fio_plot -i ./testdir/4k/ -T "TESTRUN" -d 64 -L -t iops -r randread

 Saving to file TESTRUN_2021-09-05_100527_fT.png

[root@centos1 tmp]# ~/fio-plot/fio_plot/fio_plot -i ./testdir/4k/ -T "TESTRUN" -d 128 -L -t iops -r randread
Traceback (most recent call last):
  File "/root/fio-plot/fio_plot/fio_plot", line 36, in <module>
    main()
  File "/root/fio-plot/fio_plot/fio_plot", line 29, in main
    routing_dict[item]["function"](settings, data)
  File "/root/fio-plot/fio_plot/fiolib/bar3d.py", line 90, in plot_3d
    xpos, ypos = np.meshgrid(xpos - (size / lx), ypos - (size * (ly / lx)))
ZeroDivisionError: float division by zero
[root@centos1 tmp]#

Am I doing/using something wrong?

temp.zip

[TjerkNan]

Well, I can tell you that I never tested IO depths higher than 64.

I will take a look.

Maybe I’m out off the loop regarding the current state of SSDs, but those ‘crazy’ high iodepths, do they make sense?

[louwrentius]

Would you mind putting the json files in a zip for me and attach it to this issue? Then I can test for you.

[eksklawa]

I have attached whole output folder in zip file

[louwrentius]

I have added some instructions. The default maximum iodepth is 64 and that collided with the higher iodepths. This is now shown in an error message instead of a stack trace that doesn't help much.

I also think a 2D graph is more usefull for you as numjobs is just 1 for all results. Hope it helps.