axboe / fio

Flexible I/O Tester
GNU General Public License v2.0
5.29k stars 1.26k forks source link

FIO 3.15 logs incorrect data for write_hist_log parameter #974

Open swetharajendiran opened 4 years ago

swetharajendiran commented 4 years ago

Using the write_hist_log parameter on fio versions 3.15 produce 0's at random time intervals when IO's are actually seen happening to the nvme SSD. When older fio version is used ( fio 3.5) we do not see this issue. The issue is consistently reproducible with sequential reads QD256. Input file for the workload: rd_qd_256_128k_1w.txt

Write hist with fio 3.15 - You can see the first few seconds are all 0. This happens at different points in different runs: rd_qd_256_128k_1w_clat_hist.log_clat_hist.1.log Same workload with fio 3.5 does not show this issue: rd_qd_256_128k_1w_clat_hist.log_clat_hist.1.log

vincentkfu commented 4 years ago

This should be fixed by https://github.com/axboe/fio/pull/840. The first tagged version with this fix was 3.17.

sitsofe commented 4 years ago

@swetharajendiran can you test whether master version of fio fixes your issues? Thanks!

swetharajendiran commented 4 years ago

@vincentkfu - I am hoping that fio 3.17 does not show the high latency issue (https://github.com/axboe/fio/issues/955) . I will test and confirm.

swetharajendiran commented 4 years ago

We are currently using fio 3.5 version and when I moved to fio 3.17 I saw high latencies similar to (#955) So I did a git bisect and below are the results: Git bisect for bad commit: image Git bisect log: image

sitsofe commented 4 years ago

@swetharajendiran assuming reverting just 408874d really does solve your problem can we continue this high latency discussion over in #955 ?

swetharajendiran commented 4 years ago

@sitsofe - Yes we can continue the discussion in #955