search

louwrentius / fio-plot

Create charts from FIO storage benchmark tool output
BSD 3-Clause "New" or "Revised" License
370 stars 87 forks source link

JSON files should be parsed for parameters instead of parsing the filename as with .log files #70

Closed louwrentius closed 2 years ago

louwrentius commented 2 years ago

If you want to plot charts with fio-plot, the input file names all have to adhere to a special format as documented. There is no way around this for .log files (recorded performance over time) as they don't contain any metadata.

However JSON files, by their nature and also because of the fio-plot output format, can be parsed for the variables we need, like iodepth, numjobs and the benchark type (read/write/randrw, and so on).

The key problem we would like to solve here is that from what I can tell, if files are not generated with bench-fio, mistakes are made, which in turn may lead to problems.

jsrolon commented 2 years ago

How about first greping for candidate json files which at least contain fio output keywords such as slat, clat, etc. These candidate files can then be checked against a JSON schema matching the full fio output format. If they match, they can be parsed into an in-memory JSON tree.

louwrentius commented 2 years ago

Thanks for thinking with me! I think there is no problem to just import every json we find and maybe indeed validate them against a JSON schema. In a second step, we validate that all keys are present that we require as specified on the command line.

louwrentius commented 2 years ago

Thanks a lot @jsrolon, code is merged, and a new version is released on pypi.org (1.0.17)