Add explanations for warning and ideally also for numjobs and ioqueue, remove images directory

[rubin55]

1:

See: https://github.com/louwrentius/fio-plot/blob/6b14f35c5420a5402eb032bccfb744a9c978d776/fio_plot/fiolib/dataimport.py#L211

"WARNING: the storage could not keep up with the configured I/O request size. Data is interpolated."

Can you explain what this means in detail? does it mean your data is wrong? It would be good to have a good paragraph for this, what does it mean, how does it occur, can/should you avoid it, etc.

2:

The settings iodepth and numjobs can be specified where, to my understanding, they mean:

• iodepth: how many requests fio will initiate without waiting for acknowledgement;
• numjobs: how many concurrent processes are writing and reading to (different parts of) the disk;

My personal big questionmark here is: does iodepth apply per process (numjobs) or to fio "globally"? Intuitively I would think per process, but I'm not sure and even after hours of googling I couldn't find a good explanation for it.

3:

Could the images directory not be removed or cleaned up? I ask this for two reasons:

• It contains files with characters that are not valid for filenames on Windows (:) making it annoying to copy fio-plot repository around in multi operating system environments
• It seems to me to be data created historically of a more personal nature than the project itself, the project doesn't need it + makes the repo unneccesarily large

Thanks for the great work, Just did a big performance measurement project with the help of your tools, big fan.

[louwrentius]

Thanks, those are all valid issues. I will explain some of them a bit more in depth in the readme or some other file.

Regarding could not keep up: not enough iops were issues in the fio log interval. You get this behavior from playing with floppy drives or vert slow storage devices.

Not sure if qd is per numjob, good question. I think qd is generic / global for the storage device, but I’m not sure (would make more sense to me)

3: I will look into that.

On 28 Oct 2020, at 23:00, Rubin Simons notifications@github.com wrote:

 1:

See: https://github.com/louwrentius/fio-plot/blob/6b14f35c5420a5402eb032bccfb744a9c978d776/fio_plot/fiolib/dataimport.py#L211

"WARNING: the storage could not keep up with the configured I/O request size. Data is interpolated."

Can you explain what this means in detail? does it mean your data is wrong? It would be good to have a good paragraph for this, what does it mean, how does it occur, can/should you avoid it, etc.

2:

The settings iodepth and numjobs can be specified where, to my understanding, they mean:

iodepth: how many requests fio will initiate without waiting for acknowledgement; numjobs: how many concurrent processes are writing and reading to (different parts of) the disk; My personal big questionmark here is: does iodepth apply per process (numjobs) or to fio "globally"? Intuitively I would think per process, but I'm not sure and even after hours of googling I couldn't find a good explanation for it.

3:

Could the images directory not be removed or cleaned up? I ask this for two reasons:

It contains files with characters that are not valid for filenames on Windows (:) making it annoying to copy fio-plot repository around in multi operating system environments It seems to me to be data created historically of a more personal nature than the project itself, the project doesn't need it + makes the repo unneccesarily large Thanks for the great work, Just did a big performance measurement project with the help of your tools, big fan.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[rubin55]

I did some more digging yesterday, I learned a few things, let me share my (possibly wrong, but think not) understandings:

1. blocksize == "IO Unit" size
2. iodepth is per job: when numjobs >1, fio simply clones a separate independent process (see man page). iodepth means: how many unacknowledged IO Units does a fio job (forked or not, therefore, per thread/job!) allow to exist before it starts waiting for acknowledgements again? Ergo iodepth=1 is litterally synchronous operation. This setting has nothing to do with what happens on the OS/hw layer
3. Ergo: a different blocksize can have a dramatic effect on performance, because the IO Unit (unit of work) is litterally what you allow to be asynchronous; small blocksize, say 4k with an iodepth of say 32 means: max 32 time 4k unacknowledged / in-flight, after that, wait for sync again, rinse-repeat

[louwrentius]

Thanks for sharing, this means that if you increase numjobs, the overall I/O in flight can starkly increase too, interesting!

On 29 Oct 2020, at 10:38, Rubin Simons notifications@github.com wrote:

 I did some more digging yesterday, I learned a few things, let me share my (possibly wrong, but think not) understandings:

blocksize == "IO Unit" size iodepth is per job: when numjobs >1, fio simply clones a separate independent process (see man page). iodepth means: how many unacknowledged IO Units does a fio job (forked or not, therefore, per thread/job!) allow to exist before it starts waiting for acknowledgements again? Ergo iodepth=1 is litterally synchronous operation. This setting has nothing to do with what happens on the OS/hw layer Ergo: a different blocksize can have a dramatic effect on performance, because the IO Unit (unit of work) is litterally what you allow to be asynchronous; small blocksize, say 4k with an iodepth of say 32 means: max 32 time 4k unacknowledged / in-flight, after that, wait for sync again, rinse-repeat — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[louwrentius]

I have reduced the size from ~50 MB to around 600K. The /images folder is gone, amongst other (stale) data.

[louwrentius]

I have addressed all issues. I've addressed issue 1 and 2 by updating the README or being more verbose in the messages. Issue 3 was resolved already by removing the images. Thank you for the feedback.