search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.62k stars 1.75k forks source link

MRU, MFU don't adapt to their targets #16521

Closed tkittich closed 1 month ago

tkittich commented 2 months ago

System information

Type Version/Name
Distribution Name Proxmox VE
Distribution Version pvetest
Kernel Version Linux 6.8.12-2-pve
Architecture x86_64
OpenZFS Version zfs-2.2.6-pve1 , zfs-kmod-2.2.6-pve1

Describe the problem you're observing

MRU and MFU don't change to their target sizes, especially when ARC is near its target size. It seems that arc_evict() scales the amount to get evicted proportionally to the difference between ARC's target size and current size. So when ARC is close to its target size, only a small amount of memory gets evicted. Maybe that's why MRU and MFU cannot reach their target sizes. Perhaps all the MINs in arc_evict() should be MAXs instead?

Describe how to reproduce the problem

# Limit arc max to 2GB during test
echo $((2*1024*1024*1024)) >> /sys/module/zfs/parameters/zfs_arc_max

# Before
  c     size    mfu-size  mfu-data-target   mru-size  mru-data-target   
 983M   276M      79M           80M           134M           80M   

# Write 2.1 GB file to fill ARC. Also fill MRU and change its target size
dd if=/dev/urandom of=./2.1.img bs=1M count=2150

# Check MRU, MFU sizes **every 10 minutes**. 
# Their sizes don't change much and are very different from targets.
  c     size    mfu-size  mfu-data-target   mru-size  mru-data-target   
 2.0G   2.0G      69M          741M           1.9G          741M   
 2.0G   2.0G      70M          742M           1.9G          742M   
 2.0G   2.0G      70M          743M           1.9G          743M   
 2.0G   2.0G      70M          743M           1.9G          743M   
 2.0G   2.0G      70M          743M           1.9G          743M   
 2.0G   2.0G      70M          743M           1.9G          743M
tkittich commented 2 months ago

Another example worth noting. MFU size kept decreasing instead of growing to its target. My guess is that L2ARC hits got brought into MRU and kept MRU growing instead of shrinking to its target. Stats also got updated every 10 minutes with zfs-2.2.4-pve1 , zfs-kmod-2.2.4-pve1 .

      mfusz : MFU size
     mfudat : MFU data target
      mrusz : MRU size
     mrudat : MRU data target

./arcstat.py -f c,size,read,hit%,mfusz,mfudat,mfu,mfug,mrusz,mrudat,mru,mrug,l2size,l2hits,l2hit% 600
    c   size  read  hit%  mfusz  mfudat   mfu  mfug  mrusz  mrudat   mru  mrug  l2size  l2hits  l2hit%
  51G    51G     0     0   2.7G     22G     0     0    48G     23G     0     0    654G       0       0
  51G    51G  4.6K    98   2.4G     22G  3.0K    11    48G     23G  1.5K     0    654G      17      24
  51G    51G  4.3K    98   2.5G     23G  2.8K    11    47G     23G  1.4K     0    654G      16      23
  53G    53G  7.8K    98   2.4G     25G  4.9K    16    50G     23G  2.8K     0    654G      22      16
  54G    54G  7.7K    98   2.4G     25G  4.9K    20    50G     22G  2.6K     0    654G      28      20
  53G    53G   10K    98   2.2G     26G  6.3K    17    50G     22G  3.8K     0    654G      42      24
  53G    53G  6.8K    98   2.2G     27G  4.3K    14    50G     21G  2.4K     0    654G      24      20
  53G    53G  6.6K    98   2.6G     27G  4.2K    19    50G     20G  2.2K     0    654G      37      31
  54G    53G  5.2K    98   2.4G     28G  3.4K    12    50G     20G  1.8K     0    654G      24      27
  53G    53G  6.1K    98   2.4G     28G  3.9K    13    50G     20G  2.1K     0    654G      21      21
  53G    53G  6.3K    98   2.4G     29G  4.0K    15    50G     19G  2.2K     0    654G      24      23
  53G    53G  5.2K    98   2.4G     29G  3.3K    11    50G     19G  1.8K     0    655G      18      21
  53G    53G  5.0K    98   2.3G     29G  3.2K    11    50G     18G  1.7K     0    655G      17      21
  53G    53G  8.9K    98   2.2G     30G  5.5K    19    50G     18G  3.3K     0    655G      32      21
  52G    52G  4.8K    98   2.9G     29G  3.1K     7    48G     17G  1.6K     0    655G      12      17
  51G    51G  2.4K    98   3.0G     29G  1.7K     1    47G     17G   623     0    655G       5      17
  53G    53G  5.4K    98   2.4G     31G  3.5K    14    50G     17G  1.8K     0    655G      28      31
  52G    52G  3.7K    98   2.7G     31G  2.5K    11    49G     16G  1.1K     0    655G      16      28
  51G    51G  4.5K    98   2.6G     30G  3.1K    11    48G     15G  1.4K     0    656G      18      27
  51G    51G  5.2K    98   2.4G     31G  3.6K    12    48G     15G  1.5K     0    656G      21      28
  50G    50G  5.2K    98   2.5G     31G  3.4K     7    47G     14G  1.6K     0    656G      15      20
  53G    53G  7.0K    98   2.0G     33G  4.4K    14    51G     15G  2.6K     0    656G      17      15
  54G    54G  8.8K    98   2.1G     34G  5.3K    16    51G     14G  3.4K     0    656G      21      15
  54G    54G  9.7K    98   2.0G     35G  5.7K    16    51G     13G  3.9K     0    656G      24      15
amotin commented 2 months ago

I think you may misunderstand how MRU/MFU balance works. It is based not on amount of hits or misses for the specific states, but the amount of hits for respective ghost states. I.e. if some ghost state hits predict that bigger size of respective state may be beneficial -- it will be increased. But in the first example a new file write does not produce any ghost hits, since nothing is read, so the balance is not adjusted.

In the second example you may see that mfusz is decreasing, despite mfudat growing. You haven't shown mfumet column (and you haven't added a column for total target mru/mfu, if it makes sense at all), but if we assume one also growing, then it might be that system just has not enough mfu data at all to cache, for example due to previous being evicted due to file deletion or overwrite, while all new data are mru and never have hit to promote to mfu.

tkittich commented 1 month ago

Thank you for the detailed reply. I incorrectly assumed that MRU & MFU would change to their target sizes without needing any additional ghost hits.

However, I am still seeing a strangely shrinking MFU size despite having a growing MFU target and some ghost hits. There're no file deletions or overwrites during the whole period. Would not having enough MFU data explain the shrinking size? The stats're updated every 10 minutes.

      mfudt : MFU data target
      mfuda : MFU data size
      mfumt : MFU metadata target
      mfume : MFU metadata size
      mfutg : MFU target
      mfusz : MFU size

      mrudt : MRU data target
      mruda : MRU data size
      mrumt : MRU metadata target
      mrume : MRU metadata size
      mrutg : MRU target
      mrusz : MRU size

./arcstat.py -f c,size,mfudt,mfuda,mfumt,mfume,mfutg,mfusz,mfu,mfug,mrudt,mruda,mrumt,mrume,mrutg,mrusz,mru,mrug,l2hits 600
    c   size  mfudt  mfuda  mfumt  mfume  mfutg  mfusz   mfu  mfug  mrudt  mruda  mrumt  mrume  mrutg  mrusz   mru  mrug  l2hits
  49G    49G    33G    16G   3.6G   1.3G    36G    17G   0.0   0.0   8.3G    30G   3.6G   146M    12G    31G   0.0   0.0     0.0
  48G    48G    32G    15G   3.5G   1.3G    35G    16G  2.5K   0.9   8.0G    31G   3.5G   146M    12G    31G   816   0.4      13
  48G    47G    32G    14G   3.4G   1.2G    35G    16G  2.3K   0.3   7.9G    31G   3.4G   147M    11G    31G   940   0.2      13
  47G    47G    31G    14G   3.4G   1.2G    35G    15G  2.4K   0.9   7.9G    31G   3.4G   145M    11G    31G   980   0.1      15
  46G    46G    30G    13G   3.3G   1.3G    34G    14G  2.0K   0.7   7.6G    31G   3.3G   143M    11G    31G   777   0.3     9.0
  45G    45G    30G    11G   3.3G   1.3G    33G    13G  2.1K   1.7   7.4G    31G   3.3G   145M    11G    31G   872   0.1      15
  44G    44G    30G    11G   3.2G   1.2G    33G    12G  2.1K   0.3   7.3G    31G   3.2G   143M    10G    31G   832   0.0      13
  44G    44G    29G    10G   3.2G   1.2G    32G    11G  2.1K   0.6   7.2G    31G   3.2G   143M    10G    32G   925   0.0      20
  44G    44G    29G   9.5G   3.1G   1.2G    33G    11G  1.7K   6.1   6.7G    32G   3.1G   145M   9.9G    32G   473   0.0      23
  44G    44G    30G   9.3G   3.1G   1.2G    33G    11G  1.0K   1.5   6.7G    32G   3.1G   145M   9.8G    32G   337   0.0      14
  43G    43G    29G   9.1G   3.1G   1.2G    32G    10G   768   1.6   6.5G    31G   3.1G   148M   9.5G    31G   117   0.0     4.3
  43G    43G    29G   9.0G   3.1G   1.3G    32G    10G   559   0.2   6.5G    31G   3.1G   148M   9.5G    32G   101   0.0     2.7
  43G    43G    29G   8.9G   3.1G   1.3G    32G    10G   666   0.5   6.5G    32G   3.1G   148M   9.6G    32G   177   0.1     4.3
  43G    43G    29G   8.9G   3.1G   1.3G    32G    10G   806   0.9   6.4G    32G   3.1G   150M   9.5G    32G   169   0.0     5.3
  43G    43G    29G   8.8G   3.1G   1.3G    32G    10G   765   0.1   6.4G    31G   3.1G   148M   9.4G    32G   184   0.0     4.7
  43G    43G    29G   8.6G   3.0G   1.3G    32G   9.9G  1.1K   0.2   6.3G    32G   3.0G   148M   9.4G    32G   324   0.1     8.0
  43G    43G    29G   8.7G   3.0G   1.3G    32G    10G  1.1K   0.5   6.3G    31G   3.0G   147M   9.4G    31G   288   0.1     7.0
  42G    42G    29G   8.7G   3.0G   1.3G    32G    10G   923   0.1   6.3G    31G   3.0G   145M   9.3G    31G   199   0.1     4.2
  42G    42G    29G   8.8G   3.0G   1.3G    32G    10G  1.0K   0.3   6.3G    31G   3.0G   145M   9.3G    31G   235   0.2     6.7
  42G    42G    29G   8.9G   3.0G   1.3G    32G    10G   949   0.2   6.3G    31G   3.0G   147M   9.3G    31G   231   0.2     5.4
  42G    42G    29G   9.0G   3.0G   1.3G    32G    10G   986   0.0   6.3G    31G   3.0G   144M   9.4G    31G   222   0.2     5.3
  42G    42G    29G   8.9G   3.0G   1.3G    32G    10G   732   0.1   6.3G    31G   3.0G   145M   9.3G    31G   162   0.1     3.3
  42G    42G    29G   8.7G   3.0G   1.3G    32G    10G   727   0.1   6.3G    31G   3.0G   146M   9.3G    31G   170   0.2     3.3
  42G    42G    29G   9.0G   3.0G   1.3G    32G    10G   862   0.8   6.3G    30G   3.0G   145M   9.3G    31G   161   0.2     3.9
  42G    42G    29G   8.4G   3.0G   1.3G    32G   9.7G  1.3K   0.9   6.3G    31G   3.0G   142M   9.3G    31G   494   0.4     9.7
  42G    42G    28G   8.0G   3.0G   1.3G    31G   9.3G  1.6K   0.4   6.3G    31G   3.0G   143M   9.2G    31G   587   0.4      11
  42G    42G    29G   7.7G   3.0G   1.3G    32G   9.0G  1.3K   0.2   6.3G    32G   3.0G   143M   9.4G    32G   536   0.0      11
  41G    41G    28G   6.7G   2.9G   1.3G    31G   8.0G  1.3K   0.7   6.2G    32G   2.9G   143M   9.1G    32G   481   0.0      12
  42G    42G    29G   7.4G   3.0G   1.3G    32G   8.7G   794   1.3   6.2G    32G   3.0G   144M   9.2G    32G   152   0.0     5.2
  42G    42G    29G   7.8G   3.0G   1.3G    32G   9.1G   722   0.7   6.2G    32G   3.0G   146M   9.2G    32G   132   0.0     3.4
  43G    43G    30G   6.7G   3.1G   1.3G    33G   8.0G  2.1K   0.5   6.3G    34G   3.1G   138M   9.3G    34G  1.1K   0.0     8.2
  44G    44G    31G   5.3G   3.1G   1.3G    34G   6.6G  3.0K   0.7   6.4G    37G   3.1G   139M   9.6G    37G  1.8K   0.0      12
  47G    47G    32G   4.6G   3.3G   1.3G    36G   5.9G  3.2K   0.4   6.8G    40G   3.3G   140M    10G    40G  1.9K   0.0      12
  48G    48G    33G   4.9G   3.4G   1.3G    36G   6.2G  2.1K   0.9   6.8G    40G   3.4G   142M    10G    40G   889   0.0     8.5
  47G    47G    32G   4.4G   3.3G   1.3G    36G   5.7G  1.7K   3.2   6.5G    40G   3.3G   142M   9.8G    40G   416   0.0      10
  47G    47G    33G   4.3G   3.3G   1.3G    36G   5.6G  2.1K   4.0   6.3G    40G   3.3G   141M   9.6G    40G   613   0.0      16
  47G    47G    33G   3.3G   3.3G   1.3G    36G   4.6G  2.2K   2.4   6.2G    41G   3.3G   140M   9.5G    41G   933   0.0      17
    c   size  mfudt  mfuda  mfumt  mfume  mfutg  mfusz   mfu  mfug  mrudt  mruda  mrumt  mrume  mrutg  mrusz   mru  mrug  l2hits
  47G    47G    33G   2.5G   3.3G   1.3G    37G   3.8G  3.1K   1.9   6.1G    42G   3.3G   139M   9.4G    42G  1.3K   0.0      26
  48G    48G    34G   2.7G   3.4G   1.3G    38G   4.0G  3.0K   2.3   6.2G    43G   3.4G   139M   9.6G    43G  1.2K   0.0      27
  51G    51G    36G   2.5G   3.6G   1.3G    40G   3.8G  3.9K   2.8   6.4G    46G   3.6G   138M    10G    46G  1.9K   0.1      34
  52G    52G    37G   2.7G   3.6G   1.3G    41G   4.0G  3.4K   3.1   6.4G    47G   3.6G   139M    10G    47G  1.6K   0.4      31
  51G    51G    37G   1.7G   3.6G   1.3G    40G   3.1G  2.8K   4.6   6.2G    47G   3.6G   139M   9.8G    47G  1.1K   0.7      27
  52G    52G    37G   1.8G   3.6G   1.4G    41G   3.1G  3.3K    11   6.2G    48G   3.6G   138M   9.8G    48G  1.6K   0.3      38
  52G    52G    38G   2.4G   3.6G   1.3G    41G   3.8G  3.2K    10   5.9G    47G   3.6G   138M   9.6G    47G  1.4K   0.1      34
  52G    52G    38G   1.7G   3.6G   1.3G    41G   3.0G  2.7K   3.8   5.7G    47G   3.6G   140M   9.3G    47G  1.3K   0.0      18
  52G    52G    38G   1.2G   3.6G   1.3G    41G   2.5G  2.6K   5.9   5.6G    48G   3.6G   139M   9.2G    48G  1.2K   0.0      28
  52G    52G    38G   1.7G   3.6G   1.3G    42G   3.0G  4.3K    15   5.4G    48G   3.6G   136M   9.0G    48G  2.1K   0.0      52
  52G    52G    38G   2.6G   3.6G   1.3G    42G   3.9G  3.9K   9.7   5.2G    47G   3.6G   138M   8.8G    47G  1.8K   0.0      43
  52G    52G    39G   2.0G   3.6G   1.3G    42G   3.3G  3.9K   5.3   4.9G    47G   3.6G   139M   8.5G    47G  1.9K   0.0      31
  51G    51G    38G   1.2G   3.5G   1.3G    42G   2.5G  2.7K   7.1   4.7G    48G   3.5G   139M   8.3G    48G  1.1K   0.0      20
  51G    51G    38G   1.1G   3.5G   1.4G    42G   2.4G  2.5K    16   4.6G    47G   3.5G   139M   8.0G    47G  1.1K   0.0      32
  52G    52G    39G   801M   3.5G   1.4G    43G   2.1G  3.1K   8.8   4.5G    48G   3.5G   139M   8.1G    49G  1.5K   0.0      27
  52G    52G    39G   830M   3.5G   1.3G    43G   2.1G  2.9K    14   4.4G    48G   3.5G   139M   7.9G    49G  1.5K   0.0      29
  52G    52G    39G   947M   3.5G   1.4G    43G   2.3G  2.9K    17   4.2G    48G   3.5G   139M   7.8G    49G  1.4K   0.1      36
  52G    52G    40G   664M   3.5G   1.3G    43G   2.0G  2.9K   8.5   4.1G    49G   3.5G   138M   7.7G    49G  1.5K   0.1      23
  52G    52G    40G   594M   3.5G   1.3G    43G   1.9G  3.2K    18   3.9G    48G   3.5G   138M   7.4G    49G  1.6K   0.1      32
  52G    51G    40G   1.2G   3.5G   1.4G    43G   2.6G  2.6K    14   3.7G    48G   3.5G   140M   7.2G    48G  1.3K   0.0      29
  52G    52G    40G   1.2G   3.5G   1.4G    43G   2.6G  2.5K   9.5   3.6G    48G   3.5G   142M   7.1G    48G  1.0K   0.1      24
  52G    52G    40G   1.1G   3.5G   1.4G    44G   2.5G  2.4K    14   3.5G    48G   3.5G   140M   7.0G    48G  1.1K   0.0      31
  52G    52G    41G   2.7G   3.5G   1.3G    44G   4.1G  5.5K    19   3.3G    47G   3.5G   136M   6.8G    47G  2.6K   0.2      69
  52G    52G    41G   5.3G   3.5G   1.3G    44G   6.7G  5.8K    12   3.2G    44G   3.5G   135M   6.6G    44G  2.5K   0.9      66
  52G    52G    41G   6.4G   3.4G   1.4G    44G   7.8G  5.1K   5.8   3.1G    43G   3.4G   139M   6.6G    43G  2.3K   0.6      54
  52G    52G    41G   6.2G   3.4G   1.4G    44G   7.5G  4.8K   5.5   3.0G    43G   3.4G   136M   6.5G    43G  2.2K   0.1      42
  52G    52G    41G   6.5G   3.4G   1.4G    44G   7.9G  4.8K   6.3   2.9G    43G   3.4G   134M   6.4G    43G  2.3K   0.3      41
  52G    52G    41G   6.3G   3.4G   1.4G    45G   7.7G  4.2K   4.7   2.9G    43G   3.4G   134M   6.3G    43G  2.1K   0.1      28
  52G    52G    41G   4.8G   3.4G   1.4G    44G   6.2G  3.3K   2.9   2.8G    44G   3.4G   135M   6.2G    44G  1.6K   0.1      19
  52G    52G    41G   3.1G   3.4G   1.4G    44G   4.5G  3.2K   1.8   2.8G    46G   3.4G   135M   6.2G    46G  1.8K   0.0      15
  51G    51G    41G   1.8G   3.4G   1.4G    44G   3.2G  3.1K   2.7   2.7G    47G   3.4G   136M   6.1G    47G  1.4K   0.0      15
  52G    52G    41G   797M   3.4G   1.4G    45G   2.1G  3.9K    14   2.7G    49G   3.4G   135M   6.1G    49G  2.3K   0.0      32
  52G    52G    41G   1.1G   3.4G   1.4G    45G   2.5G  3.5K    15   2.6G    48G   3.4G   135M   6.0G    48G  1.9K   0.0      30
  52G    52G    41G   939M   3.4G   1.4G    45G   2.3G  3.1K    13   2.6G    48G   3.4G   135M   5.9G    48G  1.6K   0.0      25
  52G    52G    42G   784M   3.4G   1.4G    45G   2.1G  3.4K    17   2.5G    49G   3.4G   136M   5.9G    49G  2.0K   0.0      32
  52G    52G    42G   797M   3.4G   1.4G    45G   2.2G  3.6K    15   2.4G    49G   3.4G   136M   5.8G    49G  2.1K   0.0      36
  52G    52G    42G   884M   3.4G   1.4G    45G   2.3G  3.9K    16   2.4G    48G   3.4G   134M   5.7G    49G  2.3K   0.0      37
amotin commented 1 month ago

I have no explanation for it. Either, as I have said before, something deletes the MFU data and the system just has no more of it left to cache, or it is arc_evict() wrongly evicts MFU data after not evicting MRU data. You may see that current arc_evict() logic effectively evicts MFU data for everything that left required to evict after targeted evicting from all other states. So if for some reason it won't try or can't evict MRU data, MFU data will suffer. I have no idea why would it happen, so I would recommend to try tracing arc_evict_impl() calls for amounts that was requested for eviction vs what was evicted for each state. It should show which side may misbehave.

amotin commented 1 month ago

The fact that mfudt value is growing should mean there are some MFU data ghost hits, which should mean there are some MFU data to cache. It may mean that problem is in arc_evict(), but without some additional diagnostics I can't spot it.

tkittich commented 1 month ago

However, I am still seeing a strangely shrinking MFU size despite having a growing MFU target and some ghost hits. There're no file deletions or overwrites during the whole period.

I've upgraded from zfs-2.2.4-pve1 , zfs-kmod-2.2.4-pve1 to zfs-2.2.6-pve1 , zfs-kmod-2.2.6-pve1 and still seeing similar results. Also, mfued got evected to only a few MB, while mrued remained around 40 GB.

so I would recommend to try tracing arc_evict_impl() calls for amounts that was requested for eviction vs what was evicted for each state

Thank you for looking into this problem. However, I have no idea how to trace a kernel function. ^^" If it's not too difficult and you could point me to some tutorials on how to do that, I should be able to produce the trace. 

      mfued : MFU evictable data size
      mfuem : MFU evictable metadata size
     mfugsz : MFU ghost size
      mrued : MRU evictable data size
      mruem : MRU evictable metadata size
     mrugsz : MRU ghost size

./arcstat.py -f c,size,mfudt,mfuda,mfumt,mfume,mfutg,mfusz,mfued,mfuem,mfu,mfug,mfugsz,mrudt,mruda,mrumt,mrume,mrutg,mrusz,mrued,mruem,mru,mrug,mrugsz,l2hits,l2bytes 600
    c   size  mfudt  mfuda  mfumt  mfume  mfutg  mfusz  mfued  mfuem   mfu  mfug  mfugsz  mrudt  mruda  mrumt  mrume  mrutg  mrusz  mrued  mruem   mru  mrug  mrugsz  l2hits  l2bytes
52.0G  40.9G  11.3G   4.9G   2.5G   1.2G  13.8G   6.1G   4.2G   987M   0.0   0.0    5.9G  23.1G  33.1G   2.5G   160M  25.6G  33.3G  31.0G   101M   0.0   0.0    8.0G     0.0      0.0
49.8G  49.7G  14.1G   7.3G   3.0G   1.2G  17.2G   8.5G   6.4G  1009M  5.3K   3.3    6.4G  27.9G  39.5G   3.0G   156M  31.0G  39.7G  37.7G   100M  3.2K   0.3    4.3G    42.4     5.2M
48.1G  48.1G  14.2G   3.8G   2.9G   1.2G  17.1G   5.0G   2.9G  1015M  6.1K   4.7   12.8G  26.5G  41.4G   2.9G   154M  29.4G  41.5G  39.8G  97.4M  3.2K   0.6    2.6G    38.8     4.8M
47.6G  47.6G  14.9G   1.7G   2.9G   1.2G  17.8G   2.9G   720M  1018M  5.6K  11.6   17.8G  25.4G  43.0G   2.9G   153M  28.2G  43.1G  41.4G  96.0M  2.7K   0.4    1.5G    45.7     5.2M
46.7G  46.7G  16.4G   1.1G   2.8G   1.2G  19.1G   2.3G   7.9M  1013M  5.6K  28.4   21.3G  23.2G  42.6G   2.8G   154M  26.0G  42.8G  41.3G  95.1M  2.6K   0.0    1.2G    51.2     4.6M
46.7G  46.7G  18.2G   929M   2.8G   1.2G  20.9G   2.2G  10.0M   1.0G  5.0K  25.3   22.1G  21.4G  42.8G   2.8G   156M  24.1G  42.9G  41.2G  95.2M  2.3K   0.1    1.2G    46.3     4.4M
46.5G  46.5G  19.4G   987M   2.7G   1.3G  22.1G   2.2G  14.9M   1.0G  4.8K  21.3   22.0G  20.1G  42.5G   2.7G   152M  22.8G  42.7G  41.1G  95.2M  2.4K   0.3    1.2G    45.9     4.4M
47.2G  47.2G  21.1G   1.2G   2.8G   1.3G  23.9G   2.5G   191M   1.0G  4.5K  25.9   22.2G  18.9G  42.9G   2.8G   152M  21.7G  43.1G  41.5G  94.9M  1.7K   0.2    1.3G    44.7     3.8M
47.5G  47.5G  23.5G   1.4G   2.7G   1.3G  26.2G   2.7G  69.4M   1.0G  6.0K  28.1   22.2G  16.9G  43.0G   2.7G   153M  19.6G  43.1G  41.9G  95.8M  2.2K   0.1    1.4G    40.6     3.8M
47.3G  47.2G  26.2G   1.6G   2.7G   1.3G  28.9G   2.9G   267M   1.0G  6.7K  33.4   22.0G  14.0G  42.6G   2.7G   154M  16.7G  42.7G  41.5G  96.0M  2.2K   0.3    1.5G    44.5     4.3M
47.3G  47.2G  28.2G   1.6G   2.7G   1.3G  30.9G   2.8G   305M   1.0G  5.8K  29.3   22.0G  12.1G  42.6G   2.7G   157M  14.7G  42.8G  41.4G  97.2M  2.2K   0.6    1.5G    46.6     4.1M
48.0G  48.0G  30.8G   2.2G   2.7G   1.3G  33.5G   3.5G   912M   1.0G  6.3K  26.3   22.1G  10.2G  42.7G   2.7G   156M  12.9G  42.9G  41.5G  97.9M  2.4K   0.3    1.8G    45.7     4.4M
47.5G  47.5G  32.4G   1.1G   2.6G   1.3G  35.0G   2.3G   9.1M   1.1G  6.2K  25.0   22.4G   8.2G  43.4G   2.6G   156M  10.9G  43.5G  41.9G  98.3M  2.6K   0.8    1.2G    45.1     4.5M
48.3G  48.2G  33.6G   943M   2.6G   1.3G  36.3G   2.2G  87.0M   1.1G  6.7K  27.2   22.8G   7.6G  44.2G   2.6G   153M  10.3G  44.4G  42.5G  97.7M  3.4K   0.7    1.2G    67.2     6.5M
48.4G  48.4G  34.4G   1.1G   2.6G   1.3G  37.0G   2.3G   5.3M   1.1G  5.9K  20.1   22.9G   7.1G  44.3G   2.6G   154M   9.8G  44.4G  42.8G  97.4M  3.2K   0.1    1.2G    58.6     5.9M
48.9G  48.9G  35.2G   1.1G   2.7G   1.3G  37.9G   2.4G  56.7M   1.1G  5.8K  18.9   23.0G   6.7G  44.7G   2.7G   152M   9.4G  44.9G  43.2G  96.9M  3.0K   0.1    1.3G    55.6     5.5M
52.0G  52.0G  38.2G   842M   2.8G   1.3G  41.0G   2.1G  64.2M   1.1G  7.4K  25.6   24.6G   6.5G  48.1G   2.8G   154M   9.3G  48.2G  46.1G  97.7M  4.1K   0.2    1.1G    66.2     6.3M
50.6G  50.6G  37.5G   1.2G   2.7G   1.3G  40.3G   2.5G  85.3M   1.1G  6.3K  20.2   23.8G   5.9G  46.3G   2.7G   156M   8.7G  46.4G  44.7G  99.3M  3.4K   0.1    1.3G    60.6     6.1M
49.6G  49.6G  37.3G   1.3G   2.7G   1.3G  39.9G   2.6G   5.4M   1.1G  5.9K  21.9   23.4G   5.4G  45.3G   2.7G   156M   8.1G  45.4G  43.9G  98.9M  2.9K   0.4    1.4G    59.0     6.0M
49.7G  49.7G  37.6G   1.1G   2.7G   1.3G  40.2G   2.5G  25.0M   1.1G  4.6K  15.1   23.5G   5.2G  45.4G   2.7G   156M   7.9G  45.6G  44.0G  99.1M  2.2K   0.1    1.3G    46.1     4.6M
50.0G  50.0G  38.1G   1.1G   2.7G   1.3G  40.7G   2.4G   155M   1.1G  4.8K  16.1   23.4G   5.0G  45.8G   2.7G   155M   7.6G  45.9G  44.1G  98.8M  2.4K   0.0    1.3G    42.0     4.0M
49.3G  49.3G  37.7G   991M   2.6G   1.3G  40.4G   2.3G   5.0M   1.1G  4.6K  17.3   23.3G   4.7G  45.2G   2.6G   155M   7.3G  45.3G  43.6G  98.6M  2.7K   0.1    1.2G    44.2     4.2M
49.4G  49.4G  38.1G   971M   2.6G   1.3G  40.7G   2.3G   6.5M   1.1G  4.6K  16.1   23.4G   4.5G  45.4G   2.6G   156M   7.1G  45.5G  43.8G  98.5M  2.6K   0.1    1.2G    45.9     4.5M
48.9G  48.9G  37.8G   1.2G   2.6G   1.3G  40.4G   2.5G  55.0M   1.1G  3.6K  15.6   23.0G   4.3G  44.6G   2.6G   159M   6.8G  44.7G  43.2G  98.3M  1.6K   0.0    1.3G    56.5     5.8M
45.2G  45.2G  35.0G   1.0G   2.4G   1.3G  37.4G   2.3G   177M   1.1G  4.2K  16.0   21.0G   3.8G  41.1G   2.4G   159M   6.2G  41.2G  39.6G   100M  2.2K   0.0    1.1G    51.1     5.1M
44.9G  44.9G  35.0G   1.0G   2.3G   1.3G  37.3G   2.3G  14.7M   1.1G  4.5K  15.4   21.1G   3.6G  40.8G   2.3G   161M   5.9G  40.9G  39.4G   101M  2.4K   0.0    1.2G    49.4     4.9M
46.4G  46.4G  36.3G   944M   2.4G   1.4G  38.7G   2.3G  15.1M   1.1G  3.8K  11.8   21.9G   3.6G  42.3G   2.4G   161M   6.0G  42.5G  40.8G   100M  2.0K   0.1    1.2G    45.1     4.7M
46.7G  46.7G  36.7G   852M   2.4G   1.3G  39.1G   2.2G  13.9M   1.1G  3.9K  13.2   22.1G   3.5G  42.7G   2.4G   158M   5.9G  42.9G  41.1G   100M  2.1K   0.1    1.2G    44.2     4.5M
47.0G  47.0G  37.1G   749M   2.4G   1.3G  39.5G   2.1G  45.4M   1.1G  3.3K  14.7   22.2G   3.4G  43.1G   2.4G   159M   5.8G  43.3G  41.3G   100M  1.8K   0.1    1.1G    38.2     3.5M
46.4G  46.4G  36.7G   830M   2.4G   1.3G  39.1G   2.2G   4.4M   1.1G  3.2K  14.1   22.0G   3.2G  42.5G   2.4G   158M   5.7G  42.6G  40.9G   100M  1.7K   0.1    1.2G    41.2     4.0M
45.8G  45.7G  36.3G   930M   2.4G   1.3G  38.6G   2.3G  49.7M   1.1G  3.3K  14.5   21.5G   3.1G  41.7G   2.4G   157M   5.5G  41.9G  40.2G   100M  1.8K   0.1    1.2G    42.7     4.1M
46.6G  46.6G  37.1G  1000M   2.4G   1.3G  39.5G   2.3G   3.8M   1.1G  2.3K  12.0   22.0G   3.1G  42.5G   2.4G   162M   5.5G  42.7G  41.1G   100M  1.1K   0.0    1.2G    33.7     3.1M
46.9G  46.9G  37.4G   595M   2.4G   1.3G  39.8G   1.9G  15.1M   1.1G  2.8K  12.7   22.3G   3.0G  43.3G   2.4G   160M   5.4G  43.4G  41.3G   100M  1.6K   0.0    1.0G    43.5     4.3M
46.5G  46.4G  37.1G   496M   2.4G   1.3G  39.5G   1.8G   768K   1.1G  3.3K  14.8   22.2G   2.9G  42.9G   2.4G   158M   5.3G  43.0G  40.9G   100M  2.2K   0.0  991.0M    56.2     5.7M
45.4G  45.4G  36.4G   842M   2.3G   1.4G  38.7G   2.2G  13.2M   1.1G  3.3K  14.6   21.4G   2.7G  41.4G   2.3G   164M   5.1G  41.6G  39.9G  99.4M  2.1K   0.0    1.2G    51.4     5.2M
44.4G  44.3G  35.6G   809M   2.3G   1.4G  37.8G   2.1G  46.6M   1.1G  1.5K  10.4   20.9G   2.6G  40.4G   2.3G   162M   4.9G  40.6G  38.9G  99.5M   708   0.0    1.2G    19.4     1.6M
46.5G  46.5G  37.4G   1.1G   2.4G   1.4G  39.7G   2.5G   435M   1.1G  1.5K   1.4   21.1G   2.7G  42.2G   2.4G   161M   5.1G  42.4G  40.5G   100M   849   0.0    1.3G    10.4     1.3M
    c   size  mfudt  mfuda  mfumt  mfume  mfutg  mfusz  mfued  mfuem   mfu  mfug  mfugsz  mrudt  mruda  mrumt  mrume  mrutg  mrusz  mrued  mruem   mru  mrug  mrugsz  l2hits  l2bytes
46.8G  46.7G  37.6G   908M   2.4G   1.3G  40.0G   2.2G  13.4M   1.1G  2.3K   8.4   22.1G   2.7G  42.7G   2.4G   160M   5.1G  42.9G  41.1G  99.2M  1.1K   0.0    1.2G    25.4     2.5M
46.7G  46.6G  37.6G   662M   2.4G   1.3G  40.0G   2.0G  75.4M   1.1G  2.6K  13.0   22.1G   2.6G  42.9G   2.4G   157M   5.0G  43.0G  41.0G   100M  1.3K   0.0    1.1G    31.2     2.8M
46.5G  46.5G  37.6G   770M   2.4G   1.3G  40.0G   2.1G  19.5M   1.1G  3.1K  14.3   22.1G   2.5G  42.7G   2.4G   159M   4.9G  42.8G  41.0G  99.5M  1.7K   0.1    1.1G    35.0     3.1M
46.2G  46.2G  37.5G   647M   2.3G   1.3G  39.8G   2.0G   5.1M   1.1G  2.9K  13.7   22.0G   2.4G  42.5G   2.3G   160M   4.8G  42.6G  40.7G   100M  1.6K   0.0    1.1G    32.8     2.9M
46.7G  46.7G  37.9G   739M   2.4G   1.3G  40.3G   2.0G   176M   1.1G  2.5K  11.9   21.8G   2.4G  42.8G   2.4G   159M   4.7G  43.0G  40.9G  99.3M  1.4K   0.0    1.1G    22.1     1.7M
46.9G  46.8G  38.1G   1.0G   2.4G   1.3G  40.5G   2.3G   9.7M   1.1G  1.6K  10.5   22.1G   2.4G  42.7G   2.4G   161M   4.7G  42.9G  41.2G  99.3M   781   0.0    1.2G    20.0     1.5M
46.7G  46.7G  38.0G   979M   2.4G   1.3G  40.4G   2.3G   215M   1.1G  1.8K   5.5   21.8G   2.3G  42.6G   2.4G   163M   4.7G  42.8G  40.9G  99.2M   770   0.0    1.2G    22.4     2.4M
46.9G  46.9G  38.2G   1.2G   2.4G   1.3G  40.6G   2.6G  17.8M   1.1G  1.5K   7.8   22.0G   2.3G  42.5G   2.4G   161M   4.7G  42.7G  41.3G  99.3M   636   0.0    1.3G    21.7     2.0M
46.8G  46.8G  38.2G   1.0G   2.4G   1.3G  40.5G   2.3G  25.5M   1.1G  1.6K   2.3   22.1G   2.3G  42.7G   2.4G   161M   4.6G  42.8G  41.2G  99.3M   666   0.0    1.3G    12.7     1.5M
46.9G  46.8G  38.3G   825M   2.4G   1.3G  40.6G   2.1G  10.4M   1.1G  2.3K  12.7   22.2G   2.2G  42.9G   2.4G   161M   4.6G  43.1G  41.3G  99.3M  1.1K   0.0    1.1G    29.6     2.6M
47.0G  47.0G  38.5G   823M   2.4G   1.4G  40.8G   2.2G  57.4M   1.1G  2.2K  12.3   22.2G   2.2G  43.0G   2.4G   160M   4.5G  43.2G  41.3G   100M  1.1K   0.0    1.2G    28.3     2.4M
46.5G  46.5G  38.1G   848M   2.3G   1.3G  40.4G   2.2G   6.9M   1.1G  2.0K  11.9   22.0G   2.1G  42.5G   2.3G   159M   4.4G  42.7G  40.9G  99.5M   992   0.0    1.2G    26.1     2.2M
46.5G  46.5G  38.1G   985M   2.3G   1.3G  40.5G   2.3G   2.6M   1.1G  1.9K   9.5   21.9G   2.1G  42.4G   2.3G   161M   4.4G  42.5G  40.9G  99.3M   904   0.0    1.2G    18.2     1.5M
46.7G  46.7G  38.4G   538M   2.3G   1.3G  40.7G   1.8G   4.1M   1.1G  2.6K  13.2   22.2G   2.0G  43.1G   2.3G   157M   4.3G  43.2G  41.1G   100M  1.6K   0.0  995.1M    38.3     3.7M
45.8G  45.7G  37.6G   404M   2.3G   1.3G  39.9G   1.7G   2.5M   1.1G  4.2K  15.1   21.8G   1.9G  42.2G   2.3G   157M   4.2G  42.4G  40.2G   100M  2.8K   0.0  946.9M    57.7     6.0M
46.5G  46.5G  38.3G   931M   2.3G   1.3G  40.6G   2.3G   3.0M   1.1G  1.7K   6.9   22.0G   1.9G  42.4G   2.3G   165M   4.2G  42.6G  40.9G  99.0M   825   0.0    1.2G    18.3     1.6M
46.5G  46.5G  38.4G   869M   2.3G   1.3G  40.7G   2.2G    0.0   1.1G  1.8K  11.9   22.0G   1.9G  42.5G   2.3G   161M   4.2G  42.7G  41.0G   100M   957   0.0    1.2G    29.8     2.7M
46.7G  46.7G  38.5G   1.1G   2.3G   1.3G  40.9G   2.4G   125M   1.1G   749   2.6   21.8G   1.9G  42.4G   2.3G   162M   4.2G  42.6G  41.0G  99.3M   255   0.0    1.3G     5.6   541.9K
46.8G  46.8G  38.7G   1.0G   2.3G   1.3G  41.0G   2.3G  30.6M   1.1G   985   1.8   22.1G   1.8G  42.6G   2.3G   169M   4.2G  42.8G  41.2G  98.4M   401   0.0    1.3G     3.8   409.1K
46.8G  46.8G  38.7G   1.1G   2.3G   1.3G  41.0G   2.4G  30.9M   1.1G  1.3K   2.2   22.0G   1.8G  42.5G   2.3G   163M   4.1G  42.7G  41.2G  99.4M   531   0.0    1.3G     5.6   620.4K
46.8G  46.8G  38.7G   1.2G   2.3G   1.3G  41.0G   2.5G  45.9M   1.1G  1.3K   2.5   22.0G   1.8G  42.5G   2.3G   161M   4.1G  42.6G  41.2G   100M   531   0.0    1.3G     8.1   939.2K
47.0G  47.0G  38.9G   937M   2.3G   1.3G  41.3G   2.3G  53.0M   1.1G  2.3K  11.9   22.1G   1.8G  42.9G   2.3G   161M   4.1G  43.1G  41.3G  99.5M  1.2K   0.0    1.2G    24.0     1.9M
46.9G  46.9G  39.0G   478M   2.3G   1.3G  41.3G   1.8G   4.1M   1.2G  3.4K  14.0   22.3G   1.7G  43.4G   2.3G   156M   4.0G  43.5G  41.3G   100M  2.1K   0.0  992.1M    34.2     3.1M
47.2G  47.1G  39.2G   430M   2.3G   1.3G  41.6G   1.7G   9.7M   1.2G  5.8K  17.8   22.5G   1.6G  43.6G   2.3G   155M   4.0G  43.8G  41.5G   100M  4.1K   0.0  970.9M    52.1     5.1M
48.3G  48.2G  40.3G   472M   2.4G   1.3G  42.6G   1.8G  13.9M   1.2G  6.3K  23.7   22.9G   1.6G  44.7G   2.4G   157M   4.0G  44.8G  42.5G   100M  4.4K   0.0  989.9M    54.3     4.9M
49.3G  49.3G  41.3G   658M   2.4G   1.4G  43.7G   2.0G  17.9M   1.1G  5.6K  16.5   23.5G   1.6G  45.5G   2.4G   157M   4.0G  45.7G  43.5G   100M  3.5K   0.0    1.1G    44.8     4.3M
46.6G  46.6G  38.9G   1.1G   2.3G   1.4G  41.2G   2.4G   143M   1.1G  2.5K  12.2   21.8G   1.5G  42.3G   2.3G   162M   3.7G  42.5G  40.8G  99.0M  1.2K   0.1    1.2G    25.7     2.1M
48.6G  48.6G  40.7G   1.1G   2.4G   1.4G  43.1G   2.5G   362M   1.1G  2.1K   2.2   22.4G   1.5G  44.3G   2.4G   166M   3.9G  44.4G  42.5G  98.4M  1.1K   0.0    1.3G    11.1     1.3M
48.5G  48.4G  40.6G   777M   2.4G   1.4G  43.0G   2.2G   3.5M   1.2G  2.0K  10.8   23.0G   1.5G  44.5G   2.4G   160M   3.8G  44.6G  42.7G  99.5M  1.1K   0.0    1.2G    19.2     1.4M
48.8G  48.8G  41.0G   503M   2.4G   1.4G  43.3G   1.9G  76.3M   1.2G  3.4K  13.6   23.0G   1.4G  45.1G   2.4G   158M   3.8G  45.3G  43.0G  99.3M  2.3K   0.0   1021M    21.7     1.5M
48.7G  48.6G  40.9G   565M   2.4G   1.4G  43.2G   2.0G  71.8M   1.2G  1.5K   7.1   22.9G   1.4G  44.9G   2.4G   163M   3.8G  45.0G  42.9G  98.8M   753   0.0    1.1G    13.3     1.0M
48.6G  48.6G  40.8G   492M   2.4G   1.3G  43.2G   1.8G   1.4M   1.2G  2.2K  11.6   23.0G   1.4G  44.9G   2.4G   159M   3.7G  45.1G  42.8G  99.3M  1.5K   0.0   1014M    26.3     2.2M
48.7G  48.6G  40.9G   565M   2.4G   1.4G  43.3G   1.9G  20.0M   1.2G  2.3K  11.7   23.1G   1.4G  44.9G   2.4G   160M   3.7G  45.1G  42.9G  99.1M  1.5K   0.0    1.0G    26.6     2.2M
48.6G  48.6G  40.8G   492M   2.4G   1.3G  43.2G   1.8G   1.4M   1.2G  2.2K  11.6   23.0G   1.4G  44.9G   2.4G   159M   3.7G  45.1G  42.8G  99.3M  1.5K   0.0   1014M    26.3     2.2M
48.7G  48.6G  40.9G   565M   2.4G   1.4G  43.3G   1.9G  20.0M   1.2G  2.3K  11.7   23.1G   1.4G  44.9G   2.4G   160M   3.7G  45.1G  42.9G  99.1M  1.5K   0.0    1.0G    26.6     2.2M
47.4G  47.4G  39.9G   528M   2.3G   1.4G  42.1G   1.9G   1.6M   1.2G  2.6K   9.6   22.5G   1.3G  43.7G   2.3G   160M   3.6G  43.9G  41.7G  99.3M  1.6K   0.0    1.0G    20.9     1.7M
47.5G  47.4G  40.0G   769M   2.3G   1.4G  42.2G   2.2G   5.7M   1.2G  2.7K  13.9   22.5G   1.3G  43.5G   2.3G   159M   3.6G  43.6G  41.7G  99.1M  1.6K   0.0    1.2G    30.9     2.6M
47.0G  47.0G  39.6G   854M   2.3G   1.4G  41.9G   2.2G   1.5M   1.2G  2.2K  13.2   22.3G   1.2G  43.0G   2.3G   161M   3.5G  43.1G  41.4G  98.9M  1.2K   0.0    1.2G    27.9     2.3M
45.5G  45.5G  38.3G   582M   2.2G   1.4G  40.5G   2.0G   8.4M   1.2G  3.5K  20.2   21.6G   1.2G  41.7G   2.2G   158M   3.3G  41.9G  39.9G  99.1M  2.2K   0.1    1.1G    38.8     3.1M
    c   size  mfudt  mfuda  mfumt  mfume  mfutg  mfusz  mfued  mfuem   mfu  mfug  mfugsz  mrudt  mruda  mrumt  mrume  mrutg  mrusz  mrued  mruem   mru  mrug  mrugsz  l2hits  l2bytes
45.3G  45.3G  38.2G   999M   2.2G   1.4G  40.3G   2.4G   6.0M   1.2G  2.7K  14.2   21.3G   1.1G  41.1G   2.2G   160M   3.3G  41.2G  39.7G  98.5M  1.7K   0.1    1.3G    23.9     1.8M
45.7G  45.7G  38.6G   995M   2.2G   1.4G  40.7G   2.4G  39.0M   1.2G  2.4K   9.0   21.5G   1.1G  41.5G   2.2G   159M   3.3G  41.7G  40.0G  98.8M  1.4K   0.0    1.3G    23.6     2.1M
46.1G  46.0G  38.9G   682M   2.2G   1.4G  41.1G   2.1G   6.2M   1.2G  3.6K  17.7   21.9G   1.1G  42.2G   2.2G   158M   3.3G  42.4G  40.4G  98.8M  2.2K   0.0    1.1G    33.2     2.6M
45.5G  45.5G  38.5G   623M   2.2G   1.4G  40.6G   2.0G   5.7M   1.2G  3.2K  15.2   21.6G   1.0G  41.7G   2.2G   157M   3.2G  41.8G  39.9G  97.9M  2.0K   0.0    1.1G    32.1     2.8M
46.2G  46.1G  39.1G   563M   2.2G   1.4G  41.3G   2.0G   2.9M   1.2G  2.3K  12.7   21.9G   1.0G  42.4G   2.2G   157M   3.2G  42.5G  40.5G  98.1M  1.4K   0.0    1.1G    30.6     2.7M
47.4G  47.3G  40.2G   647M   2.2G   1.4G  42.4G   2.1G   4.5M   1.2G  4.5K  20.5   22.3G   1.0G  43.5G   2.2G   155M   3.3G  43.6G  41.6G  98.3M  2.8K   0.0    1.1G    64.7     6.3M
47.1G  47.0G  40.0G   583M   2.2G   1.4G  42.2G   2.0G  10.2M   1.2G  5.5K  18.5   22.3G  1002M  43.3G   2.2G   154M   3.2G  43.4G  41.3G  98.3M  3.6K   0.1    1.1G    74.8     8.0M
46.2G  46.2G  39.3G   642M   2.2G   1.4G  41.5G   2.0G   3.5M   1.2G  4.2K  15.8   21.8G   953M  42.4G   2.2G   158M   3.1G  42.5G  40.6G  96.8M  2.6K   0.0    1.1G    44.8     4.3M
46.8G  46.7G  39.9G   925M   2.2G   1.4G  42.0G   2.3G   8.9M   1.2G  4.5K  16.8   22.1G   926M  42.6G   2.2G   158M   3.1G  42.8G  41.1G  97.1M  2.7K   0.0    1.2G    50.4     5.0M
47.0G  47.0G  40.1G   789M   2.2G   1.4G  42.3G   2.2G   9.8M   1.2G  4.1K  16.2   22.3G   898M  43.0G   2.2G   157M   3.1G  43.1G  41.3G  97.3M  2.5K   0.0    1.2G    38.7     3.5M
47.1G  47.1G  40.2G   844M   2.2G   1.5G  42.4G   2.3G  55.0M   1.2G  3.8K  14.5   22.2G   876M  43.0G   2.2G   157M   3.0G  43.1G  41.3G  97.6M  2.4K   0.0    1.2G    27.6     2.2M
46.8G  46.7G  40.0G   851M   2.2G   1.5G  42.1G   2.3G  33.2M   1.2G  3.7K  14.6   22.0G   846M  42.7G   2.2G   157M   3.0G  42.8G  41.0G  97.3M  2.2K   0.0    1.2G    30.0     2.5M
47.1G  47.1G  40.3G   751M   2.2G   1.5G  42.5G   2.2G   121M   1.2G  5.0K  15.9   22.1G   824M  43.1G   2.2G   153M   3.0G  43.3G  41.3G  97.4M  3.2K   0.0    1.1G    38.1     3.5M
46.0G  45.9G  39.3G   1.1G   2.1G   1.5G  41.4G   2.6G  24.7M   1.2G  5.3K  17.0   21.6G   775M  41.6G   2.1G   154M   2.9G  41.7G  40.2G  96.8M  3.1K   0.0    1.3G    44.4     4.2M
47.1G  47.0G  40.3G   876M   2.2G   1.5G  42.5G   2.3G  53.7M   1.2G  4.4K  14.1   22.2G   786M  42.9G   2.2G   154M   2.9G  43.0G  41.2G  96.4M  2.2K   0.1    1.2G    49.0     4.9M
47.1G  47.1G  40.4G   1.3G   2.1G   1.5G  42.5G   2.7G   348M   1.3G  3.4K   8.5   21.9G   774M  42.5G   2.1G   157M   2.9G  42.7G  41.0G  95.6M  1.9K   0.0    1.4G    30.8     3.2M
47.1G  47.0G  40.4G   1.1G   2.1G   1.4G  42.5G   2.6G  10.0M   1.3G  2.5K  11.9   22.1G   759M  42.7G   2.1G   155M   2.9G  42.8G  41.3G  95.8M  1.2K   0.0    1.3G    23.0     1.8M
47.4G  47.4G  40.7G   841M   2.2G   1.4G  42.8G   2.2G   160M   1.3G  3.0K  12.8   22.2G   746M  43.3G   2.2G   155M   2.9G  43.5G  41.4G  96.3M  1.5K   0.0    1.2G    30.3     2.7M
47.5G  47.5G  40.8G   734M   2.2G   1.4G  43.0G   2.1G  13.1M   1.3G  5.5K  17.0   22.5G   718M  43.5G   2.2G   154M   2.9G  43.7G  41.7G  96.4M  3.5K   0.0    1.1G    38.0     3.4M
47.9G  47.8G  41.2G   729M   2.2G   1.5G  43.4G   2.2G  24.1M   1.3G  4.4K  15.9   22.6G   698M  43.9G   2.2G   151M   2.8G  44.0G  42.0G  93.4M  2.4K   0.0    1.2G    42.6     4.1M
47.2G  47.2G  40.7G   699M   2.1G   1.5G  42.8G   2.1G    0.0   1.3G  4.5K  16.2   22.5G   668M  43.3G   2.1G   151M   2.8G  43.5G  41.5G  93.2M  2.6K   0.0    1.1G    38.0     3.5M
47.1G  47.1G  40.6G   588M   2.1G   1.5G  42.7G   2.1G  18.1M   1.3G  3.0K  14.9   22.4G   648M  43.2G   2.1G   154M   2.7G  43.4G  41.4G  92.4M  1.6K   0.0    1.1G    26.7     2.1M
45.4G  45.4G  39.1G   753M   2.0G   1.4G  41.1G   2.2G   2.4M   1.3G  1.8K   9.9   21.5G   615M  41.4G   2.0G   155M   2.6G  41.6G  39.8G  92.1M   957   0.0    1.2G    17.3     1.2M
44.8G  44.8G  38.6G   859M   2.0G   1.5G  40.6G   2.3G   189M   1.3G  1.6K   7.0   20.9G   601M  40.7G   2.0G   156M   2.6G  40.9G  39.0G  92.1M   828   0.0    1.2G    12.0   846.7K
46.9G  46.9G  40.6G   778M   2.1G   1.4G  42.6G   2.2G  26.8M   1.3G  4.4K  17.9   22.2G   602M  43.0G   2.1G   151M   2.7G  43.1G  41.2G  92.5M  2.4K   0.1    1.2G    35.3     3.7M
47.5G  47.4G  41.1G   441M   2.1G   1.4G  43.2G   1.9G   6.5M   1.3G  4.6K  20.9   22.4G   574M  43.8G   2.1G   150M   2.7G  43.9G  41.7G  92.7M  2.6K   0.0    1.0G    33.7     2.9M
47.4G  47.4G  41.0G   601M   2.1G   1.5G  43.1G   2.1G  12.7M   1.3G  4.4K  15.5   22.4G   559M  43.5G   2.1G   152M   2.6G  43.7G  41.6G  92.4M  2.6K   0.0    1.1G    33.3     2.9M
47.2G  47.2G  40.9G   818M   2.1G   1.5G  43.0G   2.3G    0.0   1.3G  4.0K  16.5   22.4G   540M  43.2G   2.1G   152M   2.6G  43.4G  41.5G  92.7M  2.3K   0.0    1.2G    36.5     3.2M
amotin commented 1 month ago

I meant using dtrace/bpftrace to log the function input and output. But it seems like you've already caught it at https://github.com/openzfs/zfs/pull/16546 .