ZFS 0.8.4 (from source) giving very inaccurate ETA for resilver while attaching larger disk to mirror

[kneutron]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	19.04 64-bit
Linux Kernel	5.3.0-55-generic
Architecture	Quad-core Intel Core i5-6400 CPU @ 2.70GHz, 32 GB RAM, ASUS Z170 Deluxe mobo, not overclocked, booting in non-UEFI mode
ZFS Version	0.8.4 compiled from source
SPL Version	0.8.4

Describe the problem you're observing

4x4TB disk mirror pool, attaching 2x6TB disks one at a time to mirror-0 to increase pool size with no degradation, with the aim of detaching the 2 original 4TB disks afterward to be repurposed

' zpool status; zpool iostat ' shows very inaccurate estimate as to % done and ETA ( resilver had been running for ~15 minutes with " echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay " to speed up resilver I/O. Scrub immediately preceding this operation took:

scan: scrub repaired 0B in 0 days 08:52:42 with 0 errors on Sat Jun 20 22:01:42 2020

Disks are connected via SAS with breakout cables:

Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

Most datasets are recordsize=1M but the pool is not updated with all current features: (this pool can probably be upgraded since I don't intend on going back to any previous ZFS versions and it will stay on Linux)

zpool upgrade

This system supports ZFS pool feature flags.

All pools are formatted using feature flags.

Some supported features are not enabled on the following pools. Once a
feature is enabled the pool may become incompatible with software
that does not support the feature. See zpool-features(5) for details.

POOL  FEATURE
---------------
zseatera4
      multi_vdev_crash_dump
      large_dnode
      sha512
      skein
      edonr
      userobj_accounting
      encryption
      project_quota
      device_removal
      obsolete_counts
      zpool_checkpoint
      spacemap_v2
      allocation_classes
      resilver_defer
      bookmark_v2

Pool: zseatera4 - resilver started: Sat 20 Jun 2020 10:14:55 PM CDT
  scan: resilver in progress since Sat Jun 20 22:14:52 2020
        5.60T scanned at 5.29G/s, 2.93T issued at 2.77G/s, 5.60T total
        131G resilvered, 52.34% done, 0 days 00:16:27 to go
                                         capacity     operations     bandwidth
pool                                   alloc   free   read  write   read  write
-------------------------------------  -----  -----  -----  -----  -----  -----
zseatera4                              5.60T  1.65T    166    162   162M   158M
  mirror                               2.79T   850G    163    162   162M   158M
    ata-ST4000VN000-2AH166                 -      -     67      0  66.5M      0
    ata-ST4000VN000-1H4168                 -      -     96      0  95.9M      0
    ata-HGST_HUS726060ALE614               -      -      0    162      0   158M
  mirror                               2.80T   844G      2      0  22.0K      0
    ata-ST4000VN000-1H4168                 -      -      0      0  7.99K      0
    ata-ST4000VN008-2DR166                 -      -      1      0  14.0K      0
-------------------------------------  -----  -----  -----  -----  -----  -----
                                         capacity     operations     bandwidth
pool                                   alloc   free   read  write   read  write
-------------------------------------  -----  -----  -----  -----  -----  -----
zseatera4                              5.60T  1.65T    176    178   172M   174M
  mirror                               2.79T   850G    174    178   172M   174M
    ata-ST4000VN000-2AH166                 -      -     70      0  69.4M      0
    ata-ST4000VN000-1H4168                 -      -    104      0   102M      0
    ata-HGST_HUS726060ALE614               -      -      0    178      0   174M
  mirror                               2.80T   844G      1      0  14.0K      0
    ata-ST4000VN000-1H4168                 -      -      0      0  7.99K      0
    ata-ST4000VN008-2DR166                 -      -      0      0  5.99K      0
-------------------------------------  -----  -----  -----  -----  -----  -----

Updated at 22:50 below:

Pool: zseatera4 - resilver started: Sat 20 Jun 2020 10:14:55 PM CDT
  scan: resilver in progress since Sat Jun 20 22:14:52 2020
        5.60T scanned at 2.66G/s, 3.09T issued at 1.47G/s, 5.60T total
        296G resilvered, 55.21% done, 0 days 00:29:08 to go

Describe how to reproduce the problem

Attach a larger disk to a ZFS RAID10 mirror-0, one at a time and wait for resilver to finish The pool is currently:

# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zseatera4  7.25T  5.60T  1.65T        -         -    22%    77%  1.00x    ONLINE  -

Include any warning/errors/backtraces from the system logs (N/A)

[kneutron]

Bump - this appears to be a regression, ETA was not this bad in previous releases

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[ghost]

See also #11779