switch of chassis (with ongoing scrub) metadata lost

[zeph]

Hi, I just moved some drives from a box to another, all good, zpool import -a worked Today, it comes back with this... (we are on Catalina .... driver 1.9.3.1 and 1.9.2 ...was 1.9.3.0)

   pool: magodo
     id: 10910979000043290031
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

    magodo                                          FAULTED  corrupted data
      raidz2-0                                      FAULTED  corrupted data
        media-016F4BB7-4850-9041-A4B2-4A8A9391544C  ONLINE
        media-BD133939-C13E-F041-8CBF-208395F97DF8  ONLINE
        media-113BE775-71D0-E448-957B-D6E6C25015BD  ONLINE
        media-02BAAAF5-0FE3-5442-AE88-9CBDC3D5B0AE  ONLINE
        media-5F83C370-7BCC-BC47-8AD1-F224669488CA  ONLINE

# zpool import -f  magodo
internal error: Malformed Mach-o file
Abort trap: 6

spindump.txt

[zeph]

# zpool import -N -oreadonly -T 1663455 
   pool: magodo
     id: 10910979000043290031
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

    magodo                                          FAULTED  corrupted data
      raidz2-0                                      FAULTED  corrupted data
        media-016F4BB7-4850-9041-A4B2-4A8A9391544C  ONLINE
        media-BD133939-C13E-F041-8CBF-208395F97DF8  ONLINE
        media-113BE775-71D0-E448-957B-D6E6C25015BD  ONLINE
        media-02BAAAF5-0FE3-5442-AE88-9CBDC3D5B0AE  ONLINE
        media-5F83C370-7BCC-BC47-8AD1-F224669488CA  ONLINE

[zeph]

# zpool import -N -oreadonly -T 1663446
   pool: magodo
     id: 10910979000043290031
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:

    magodo                                          FAULTED  corrupted data
      raidz2-0                                      ONLINE
        media-016F4BB7-4850-9041-A4B2-4A8A9391544C  ONLINE
        media-BD133939-C13E-F041-8CBF-208395F97DF8  ONLINE
        media-113BE775-71D0-E448-957B-D6E6C25015BD  ONLINE
        media-02BAAAF5-0FE3-5442-AE88-9CBDC3D5B0AE  ONLINE
        media-5F83C370-7BCC-BC47-8AD1-F224669488CA  ONLINE

[zeph]

# zpool import -f -N -oreadonly -T 1663441 magodo
cannot import 'magodo': one or more devices is currently unavailable

[zeph]

sudo dtrace -qn 'zfs_dbgmsg_mac:entry{printf("%s\n", stringof(arg0));}' from @lundman

[zeph]

spa_tryimport: importing magodo, max_txg=1663426
spa_load($import, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa $import. txg 1663426
spa_load($import, config untrusted): using uberblock with txg=1663426
spa_load($import, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load($import, config untrusted): UNLOADING
spa_tryimport: importing magodo, max_txg=1663426
spa_load($import, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa $import. txg 1663426
spa_load($import, config untrusted): using uberblock with txg=1663426
spa_load($import, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load($import, config untrusted): UNLOADING
spa_import: importing magodo, max_txg=1663426 (RECOVERY MODE)
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663426
spa_load(magodo, config untrusted): using uberblock with txg=1663426
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663425
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663425
spa_load(magodo, config untrusted): using uberblock with txg=1663425
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=88]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663424
spa_load(magodo, config untrusted): LOADING
spa_load(magodo, config untrusted): FAILED: no valid uberblock found
spa_load(magodo, config untrusted): UNLOADING

[zeph]

second run on initial 1663456

spa_import: importing magodo, max_txg=1663456 (RECOVERY MODE)
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663456
spa_load(magodo, config untrusted): using uberblock with txg=1663456
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663455
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663455
spa_load(magodo, config untrusted): using uberblock with txg=1663455
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663454
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663454
spa_load(magodo, config untrusted): using uberblock with txg=1663454
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663453
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663453
spa_load(magodo, config untrusted): using uberblock with txg=1663453
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663452
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663452
spa_load(magodo, config untrusted): using uberblock with txg=1663452
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663451
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663451
spa_load(magodo, config untrusted): using uberblock with txg=1663451
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663450
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663450
spa_load(magodo, config untrusted): using uberblock with txg=1663450
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663449
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663449
spa_load(magodo, config untrusted): using uberblock with txg=1663449
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663448
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663448
spa_load(magodo, config untrusted): using uberblock with txg=1663448
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663447
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663447
spa_load(magodo, config untrusted): using uberblock with txg=1663447
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663446
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663446
spa_load(magodo, config untrusted): using uberblock with txg=1663446
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663445
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663445
spa_load(magodo, config untrusted): using uberblock with txg=1663445
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663444
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663444
spa_load(magodo, config untrusted): using uberblock with txg=1663444
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663443
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663443
spa_load(magodo, config untrusted): using uberblock with txg=1663443
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663442
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663442
spa_load(magodo, config untrusted): using uberblock with txg=1663442
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663441
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663441
spa_load(magodo, config untrusted): using uberblock with txg=1663441
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663440
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663440
spa_load(magodo, config untrusted): using uberblock with txg=1663440
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663439
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663439
spa_load(magodo, config untrusted): using uberblock with txg=1663439
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663438
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663438
spa_load(magodo, config untrusted): using uberblock with txg=1663438
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663437
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663437
spa_load(magodo, config untrusted): using uberblock with txg=1663437
spa_load(magodo, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663436
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663436
spa_load(magodo, config untrusted): using uberblock with txg=1663436
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663435
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663435
spa_load(magodo, config untrusted): using uberblock with txg=1663435
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663434
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663434
spa_load(magodo, config untrusted): using uberblock with txg=1663434
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663433
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663433
spa_load(magodo, config untrusted): using uberblock with txg=1663433
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663432
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663432
spa_load(magodo, config untrusted): using uberblock with txg=1663432
spa_load(magodo, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663431
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663431
spa_load(magodo, config untrusted): using uberblock with txg=1663431
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663430
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663430
spa_load(magodo, config untrusted): using uberblock with txg=1663430
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663429
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663429
spa_load(magodo, config untrusted): using uberblock with txg=1663429
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663428
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663428
spa_load(magodo, config untrusted): using uberblock with txg=1663428
raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=88]
raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=88]
spa_load(magodo, config trusted): FAILED: vdev_load failed [error=88]
spa_load(magodo, config trusted): UNLOADING
spa_load(magodo, config trusted): spa_load_retry: rewind, max txg: 1663427
spa_load(magodo, config trusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663427
spa_load(magodo, config untrusted): using uberblock with txg=1663427
spa_load(magodo, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663426
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663426
spa_load(magodo, config untrusted): using uberblock with txg=1663426
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663425
spa_load(magodo, config untrusted): LOADING
disk vdev '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C': best uberblock found for spa magodo. txg 1663425
spa_load(magodo, config untrusted): using uberblock with txg=1663425
spa_load(magodo, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=88]
spa_load(magodo, config untrusted): UNLOADING
spa_load(magodo, config untrusted): spa_load_retry: rewind, max txg: 1663424
spa_load(magodo, config untrusted): LOADING
spa_load(magodo, config untrusted): FAILED: no valid uberblock found
spa_load(magodo, config untrusted): UNLOADING

[zeph]

disks are back in the old enclosure... can't import either

[zeph]

this looks familiar... https://github.com/zfsonlinux/zfs/issues/2452#issuecomment-61020758

[zeph]

more interesting stuff from https://github.com/zfsonlinux/zfs/issues/7808#issuecomment-415187446

root@hagrid:~# mkdir dumpdir

root@hagrid:~# zdb -dep /dev/disk/by-id -G magodo -x dumpdir 
zdb: can't open 'magodo': Invalid exchange

ZFS_DBGMSG(zdb) START:
spa.c:5475:spa_import(): spa_import: importing magodo
spa_misc.c:408:spa_load_note(): spa_load(magodo, config trusted): LOADING
vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/wwn-0x5000c500b2b40d9e-part1': best uberblock found for spa magodo. txg 1663456
spa_misc.c:408:spa_load_note(): spa_load(magodo, config untrusted): using uberblock with txg=1663456
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 7205757987488084710: path changed from '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C' to '/dev/disk/by-id/wwn-0x5000c500b2b40d9e-part1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 18392482608887801473: path changed from '/private/var/run/disk/by-id/media-BD133939-C13E-F041-8CBF-208395F97DF8' to '/dev/disk/by-id/wwn-0x5000c500b318b442-part1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 16699662544002398418: path changed from '/private/var/run/disk/by-id/media-113BE775-71D0-E448-957B-D6E6C25015BD' to '/dev/disk/by-id/wwn-0x5000c500b319878d-part1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 194297530426853465: path changed from '/private/var/run/disk/by-id/media-02BAAAF5-0FE3-5442-AE88-9CBDC3D5B0AE' to '/dev/disk/by-id/wwn-0x5000c500b2b8aa8c-part1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 10780039664799416240: path changed from '/private/var/run/disk/by-id/media-5F83C370-7BCC-BC47-8AD1-F224669488CA' to '/dev/disk/by-id/wwn-0x5000c500b2b6b10e-part1'
vdev.c:129:vdev_dbgmsg(): raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=52]
vdev.c:129:vdev_dbgmsg(): raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=52]
spa_misc.c:393:spa_load_failed(): spa_load(magodo, config trusted): FAILED: vdev_load failed [error=52]
spa_misc.c:408:spa_load_note(): spa_load(magodo, config trusted): UNLOADING
ZFS_DBGMSG(zdb) END

root@hagrid:~# tree dumpdir/
dumpdir/
├── wwn-0x5000c500b2b40d9e-part1
├── wwn-0x5000c500b2b6b10e-part1
├── wwn-0x5000c500b2b8aa8c-part1
├── wwn-0x5000c500b318b442-part1
└── wwn-0x5000c500b319878d-part1

0 directories, 5 files

[lundman]

Ah it is a shame that the old enclosure didn't work. See if you can entice the Linux guys to give a hand, they have hundred of devs :)

[zeph]

# zpool import -d dumpdir/ 
   pool: magodo
     id: 10910979000043290031
  state: UNAVAIL
 status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://zfsonlinux.org/msg/ZFS-8000-EY
 config:

    magodo                      UNAVAIL  insufficient replicas
      raidz2-0                  UNAVAIL  insufficient replicas
        wwn-0x5000c500b2b40d9e  UNAVAIL
        wwn-0x5000c500b318b442  UNAVAIL
        wwn-0x5000c500b319878d  UNAVAIL
        wwn-0x5000c500b2b8aa8c  UNAVAIL
        wwn-0x5000c500b2b6b10e  UNAVAIL

[zeph]

@lundman I'm posting on the channel... let's see

[zeph]

# zdb -e magodo

Configuration for import:
        vdev_children: 1
        version: 5000
        pool_guid: 10910979000043290031
        name: 'magodo'
        state: 0
        hostid: 948055264
        hostname: ''
        vdev_tree:
            type: 'root'
            id: 0
            guid: 10910979000043290031
            children[0]:
                type: 'raidz'
                id: 0
                guid: 11109210761103586910
                nparity: 2
                metaslab_array: 41
                metaslab_shift: 36
                ashift: 12
                asize: 10001923440640
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 7205757987488084710
                    whole_disk: 1
                    DTL: 69
                    create_txg: 4
                    path: '/dev/sdd1'
                    devid: 'ata-ST2000DM006-2DM164_Z560WTAB-part1'
                    phys_path: '/dev/disk/by-uuid/10910979000043290031'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 18392482608887801473
                    whole_disk: 1
                    DTL: 68
                    create_txg: 4
                    path: '/dev/sde1'
                    devid: 'ata-ST2000DM006-2DM164_Z4ZC9BF9-part1'
                    phys_path: '/dev/disk/by-uuid/10910979000043290031'
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 16699662544002398418
                    whole_disk: 1
                    DTL: 67
                    create_txg: 4
                    path: '/dev/sda1'
                    devid: 'ata-ST2000DM006-2DM164_Z4ZC9HC1-part1'
                    phys_path: '/dev/disk/by-uuid/10910979000043290031'
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 194297530426853465
                    whole_disk: 1
                    DTL: 66
                    create_txg: 4
                    path: '/dev/sdb1'
                    devid: 'ata-ST2000DM006-2DM164_Z560WKSH-part1'
                    phys_path: '/dev/disk/by-uuid/10910979000043290031'
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 10780039664799416240
                    whole_disk: 1
                    DTL: 65
                    create_txg: 4
                    path: '/dev/sdc1'
                    devid: 'ata-ST2000DM006-2DM164_Z560WWDF-part1'
                    phys_path: '/dev/disk/by-uuid/10910979000043290031'
        load-policy:
            load-request-txg: 18446744073709551615
            load-rewind-policy: 2
zdb: can't open 'magodo': Invalid exchange

ZFS_DBGMSG(zdb) START:
spa.c:5475:spa_import(): spa_import: importing magodo
spa_misc.c:408:spa_load_note(): spa_load(magodo, config trusted): LOADING
vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/sdd1': best uberblock found for spa magodo. txg 1663456
spa_misc.c:408:spa_load_note(): spa_load(magodo, config untrusted): using uberblock with txg=1663456
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 7205757987488084710: path changed from '/private/var/run/disk/by-id/media-016F4BB7-4850-9041-A4B2-4A8A9391544C' to '/dev/sdd1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 18392482608887801473: path changed from '/private/var/run/disk/by-id/media-BD133939-C13E-F041-8CBF-208395F97DF8' to '/dev/sde1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 16699662544002398418: path changed from '/private/var/run/disk/by-id/media-113BE775-71D0-E448-957B-D6E6C25015BD' to '/dev/sda1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 194297530426853465: path changed from '/private/var/run/disk/by-id/media-02BAAAF5-0FE3-5442-AE88-9CBDC3D5B0AE' to '/dev/sdb1'
vdev.c:2084:vdev_copy_path_impl(): vdev_copy_path: vdev 10780039664799416240: path changed from '/private/var/run/disk/by-id/media-5F83C370-7BCC-BC47-8AD1-F224669488CA' to '/dev/sdc1'
vdev.c:129:vdev_dbgmsg(): raidz-0 vdev (guid 11109210761103586910): unable to read the metaslab array [error=52]
vdev.c:129:vdev_dbgmsg(): raidz-0 vdev (guid 11109210761103586910): vdev_load: metaslab_init failed [error=52]
spa_misc.c:393:spa_load_failed(): spa_load(magodo, config trusted): FAILED: vdev_load failed [error=52]
spa_misc.c:408:spa_load_note(): spa_load(magodo, config trusted): UNLOADING
ZFS_DBGMSG(zdb) END

[zeph]

worth a mention https://www.klennet.com/zfs-recovery/default.aspx

[JMoVS]

What you can try what helped me (I was stuck with a pool expecting a certain cache device and refusing to import) is to generate symlinks to the paths that zdb shows. So if the pool devices were previously imported with /var/run/disk/by-id and you see those paths in youe zdb, recreate it with symlinks to the block devices at the specific place in the fs. I did that and was able to import then.

[JMoVS]

So try recreating the exact original path it is expecting according to the debug info and if the pool imports but doesn‘t show datasets, that can also happen. For me, zfs list didn‘t work but zfs mount dataset name (read of zdb or my brain) worked so I could get the data out...

[zeph]

@JMoVS already attempted tonight... :/

tnx

[zeph]

trying out ZFS Recover and UFS Explorer Professional Recovery

any previous experience?

[zeph]

ok, I might try with this https://github.com/zfsonlinux/zfs/issues/2452#issuecomment-61020758

[zeph]

ok... i spent and recovered my data... guys, not good... (I had better expectations of ZFS)