`unable to read the metaslab array [error=5]` while importing the pool

[kandeshvari]

Maybe wrong place to ask this, but give me hope...

FreeBSD 12.1-p8

I had mirror on two geli partitions. I have replaced one disk with bigger one, resilvered it completely but I mixed up with geli encryption keys. After reboot new vdev didn't attached, so the pool was running on the one old vdev. I have detached new vdev for fixing geli keys issue. Fixed it and reboot again, but I forgot to attach new vdev back to pool! After reboot this pool has switched to FAILURE state and doesn't want to import now.

I've tried to do some magic with zpool import -Ffm zroot, but it fails:

cannot import 'zroot': I/O error
        Destroy and re-create the pool from
        a backup source.

I tried to use zdb -e zroot for debugging, but it fails also with:

zdb: can't  open 'zroot': Input/output error

...debug messages...
disk vdev '/dev/ada0p4.eli': best uberblock found for spa zroot. txg 26002478
disk vdev '/dev/ada0p4.eli': unable to read the metaslab array [error=5]
disk vdev '/dev/ada0p4.eli': vdev_load: metaslab_init failed [error=5]
...

So I can't import pool and can't add new vdev to exported pool (if it possible - I don't know how to do it).

In the result, I have one broken pool on one vdev and two detached vdevs with recent data (old vdev and new one).

My questions:

1. How to fix broken pool? ))
2. Is it possible to add second, good vdev to the exported pool and try to import it again?
3. Can I assemble new pool with data from old pool and those two detached disks with data? Maybe some tools for-internal-use exists?
4. If all above is negative - where to start to learn zfs internals to debug my issue thoroughly, and create tool that would create "Frankenstain's pool" from these three disks?

Any help will be appreciated

[allanjude]

Can you summarize the state of each disk.

I expect you should be able to import the pool using one of the 2 detached disks (although this version of the pool will be behind from the time where it was detached).

Did something happen to the ada0p4.eli vdev? I don't understand how it might have been damaged if you only detached ada1p4.eli etc.

[kandeshvari]

Thank you for answering, Allan.

Let me describe all steps I done

• There were two vdevs - ada0p4.eli and ada1p4.eli.
• I wanted to replace ada1p4.eli with ada2p4.eli.
• I added ada2p4.eli to the pool, resilvered it completely and detached ada1p4.eli from the pool.
• I ran zpool online -e on the pool also. So, ada1p4.eli has all original data unchanged.
• After that I reboot. I mixed up the passwords for ada2p4.eli, and it wasn't attached to the pool.
• At this point, pool was in degraded state with ONLINE ada0p4.eli and ada2p4.eli FAILED (with strange long digit-only name).
• Next, I detached ada2p4.eli for geli maintenance.
• After fixing the geli password/key I rebooted (without attach it back). So, ada2p4.eli also has full copy of my data as well as ada1p4.eli, but with extended space.
• After reboot happened what I described in original message.

In the result:

• I think it's not a geli problem
• two detached vdevs have full copy of my data (with little differences, but it's ok)
• ada0p4.eli was in ONLINE state, when I detached ada2p4.eli and no other issues happened with it
• SMART shows all disks have no errors at all

You have mentioned there is a possibility to extract pool metadata from the detached disks. If I understand you right - how to do it?

[kandeshvari]

I have solved it. I recovered old pool from detached mirrored vdev.

For future searchers: https://gist.github.com/kandeshvari/6d6a267263585123009c98120957131f