search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.65k stars 1.75k forks source link

From 0.8.4 to 2.0.0 on Ubuntu 18 - prefetech broke #11344

Closed mickg10 closed 2 years ago

mickg10 commented 3 years ago

Using zfs 0.8.4 on Ubuntu 18, had low read performance on big sequential reads - turned out to be a prefetch issue.

Fixed it by doing:

options zfs l2arc_write_max=134217728
options zfs l2arc_noprefetch=1
options zfs zfs_txg_synctime_ms=15000
options zfs zfs_arc_max=15032385536
options zfs zfs_pd_bytes_max=1073741824
options zfs zfetch_max_distance=1073741824
options zfs zfetch_array_rd_sz=1073741824

After upgrading to ZFS 2.0.0, the prefetch behavior broke again during all sequential reads: cat $BENCHDIR/result.mov | pv > /dev/null

Results in: a 4 seconds of prefetch misses followed by several seconds of prefetch hits, followed by 0 prefetch hits afterward for the duration of the file read.

arcstat -f "time,hit%,dh%,dhit,phit,pmis,ph%,mh%" 1
    time  hit%  dh%  dhit  phit  pmis  ph%  mh%
06:30:16   100  100    15     0     0    0  100
06:30:17     0    0     0     0     0    0    0
06:30:18     0    0     0     0     0    0    0
06:30:19    57   99   10K   559  8.6K    6   99
06:30:20    74   99   12K     0  4.2K    0   99
06:30:21    75   99   21K     0  7.1K    0   99
06:30:22    77   99   18K     0  5.5K    0   99
06:30:23    75   99   17K   544  6.0K    8   99
06:30:24    86   99   15K  2.4K  2.7K   47   99
06:30:25    93   99  5.6K  1.5K   468   75   99
06:30:26    93  100  1.3K   347   115   75  100
06:30:27    93   99  2.0K   600   185   76   99
06:30:28    96  100  1.0K   292    50   85  100
06:30:29    95   99  1.2K   369    67   84   99
06:30:30    96  100  1.1K   346    46   88  100
06:30:31    94   99  2.2K   724   179   80   99
06:30:32    96   99  2.6K   899   136   86   99
06:30:33    98   99  2.8K   938    56   94   99
06:30:34    98   99   11K  3.8K   178   95   99
06:30:35    99   99   10K  3.4K    66   98   99
06:30:36    99   99   31K   10K     0  100   99
06:30:37    99   99   31K   10K     0  100   99
06:30:38    86   99   24K  2.9K  4.4K   39   99
06:30:39    78   99   22K     0  6.1K    0   99
06:30:40    68   99   17K     0  8.2K    0   99

iostat is showing at around 50%:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.15    0.00   17.46    1.60    0.00   80.79

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdr              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sds              0.00    1.00      0.00     10.00     0.00     1.50   0.00  60.00    0.00    2.00   0.00     0.00    10.00   2.00   0.20
sdq              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
sdd            112.00    0.00  54664.25      0.00   143.25     0.00  56.12   0.00   18.04    0.00   2.03   488.07     0.00   5.33  59.70
sdb            174.75    0.00  54243.25      0.00   192.50     0.00  52.42   0.00   14.44    0.00   2.53   310.40     0.00   3.65  63.80
sdc            169.75    0.00  53976.37      0.00   216.75     0.00  56.08   0.00   14.86    0.00   2.56   317.98     0.00   3.23  54.90
sde            180.25    0.00  53140.25      0.00   209.75     0.00  53.78   0.00   11.73    0.00   2.13   294.81     0.00   2.65  47.80
sdf            169.00    0.00  53248.75      0.00   189.50     0.00  52.86   0.00   14.38    0.00   2.44   315.08     0.00   3.53  59.70
sdg            193.50    0.00  53468.88      0.00   180.00     0.00  48.19   0.00   11.84    0.00   2.30   276.32     0.00   2.73  52.90
sdl            131.75    0.00  53743.00      0.00   177.50     0.00  57.40   0.00   23.69    0.00   3.13   407.92     0.00   5.43  71.50
sdn            162.75    0.00  53098.25      0.00   204.75     0.00  55.71   0.00   16.26    0.00   2.68   326.26     0.00   3.73  60.70
sdi            121.50    0.00  52699.12      0.00   142.00     0.00  53.89   0.00   15.24    0.00   1.88   433.74     0.00   4.56  55.40
sdh            158.25    0.00  54339.00      0.00   205.00     0.00  56.43   0.00   15.95    0.00   2.54   343.37     0.00   3.53  55.90
sdk            126.25    0.00  54747.13      0.00   148.75     0.00  54.09   0.00   15.61    0.00   2.01   433.64     0.00   4.82  60.80
sdm            169.75    0.00  55003.38      0.00   193.50     0.00  53.27   0.00   14.14    0.00   2.40   324.03     0.00   3.20  54.30
sdj            141.75    0.00  53429.75      0.00   142.50     0.00  50.13   0.00   11.70    0.00   1.68   376.93     0.00   3.54  50.20
sdp            155.50    0.00  54107.62      0.00   202.25     0.00  56.53   0.00   15.69    0.00   2.46   347.96     0.00   3.31  51.50
sdo            125.00    0.00  54286.50      0.00   157.00     0.00  55.67   0.00   14.90    0.00   1.89   434.29     0.00   4.64  58.00

Is there any new way to trouble-shoot what prefetch is doing and why it is not working?

stale[bot] commented 2 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.