zPool do not import file vdev on boot-time

captain-haddock17 commented 4 years ago

Hello,

Short story

Pools created with a file vdev are not imported at startup (and zfs dataset are not mounted -- de facto) .
pool are correctlty recorded in zfs.cache
zpool import at macOS startup do not reference zfs.cache, but medias referenced in /var/run/disk/by-id

Clue

zPool created with a file vdev is not referenced in the default /var/run/disk/by-id used at startup to import pools.

I tryied to make a ln -s /vdev/zFile.dat /var/run/disk/by-id/media-554310920175130116 ; but this soft-link seems to be wiped-out by MacOS at early startup.

Note : FreeBSD is ok with mounting thoses pools created with a file vdev, because it uses the zfs.cache at boot-time.

Config MacBookAir (2012) with SSD (APFS) - macOS 10.14.6 (18G4032)/ Darwin 18.7.0 sysctl {spl,zfs}.kext_version

spl.kext_version: 1.9.4-0 zfs.kext_version: 1.9.4-0

Work around

(tbd)

ZFS working environment

[MacOS] zdb -C (partial output)

zTest: version: 5000 name: 'zTest' state: 0 txg: 11798 com.delphix:has_per_vdev_zaps vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 554310920175130116 create_txg: 4 children[0]: type: 'file' id: 0 guid: 6816228191707103066 path: '/vdev/zTest.pool'

Test case

Both on MacOS and FreeBSD

Create zPool sudo mkdir /vdev sudo dd bs=4096 count=131072 if=/dev/zero of=/vdev/zTest.dat sudo zpool create zTest /vdev/zTest.dat`
Verify import/export functionality sudo zpool export zTest sudo zpool import zTest zpool list zTest
reboot
list zpool list zTest

(none)
Force import trhough commandline sudo zpool import zTest

(none) sudo zpool import -d /vdev 554310920175130116 (OK)

lundman commented 4 years ago

I'm actually surprised that freebsd will import file-based pools at all, didn't think any platform did. You will have to add a line to the the zpool-import script to add "-d /path/to/pool"

captain-haddock17 commented 4 years ago

Hello Jurgen,

I just rebooted my MacBook Air with new startup params : import -c /etc/zfs/zpool.cache in file /usr/local/libexec/zfs/launchd.d/zpool-import-all.sh

No better result, pools aren't imported

after login : zpool list

no pools available

To sumarize : none of these works. I guess problem is else-where

# "${ZPOOL}" import -a -d /var/run/disk/by-id # "${ZPOOL}" import -d /vdev 554310920175130116 "${ZPOOL}" import -c /etc/zfs/zpool.cache

captain-haddock17 commented 4 years ago

I'm actually surprised that freebsd will import file-based pools at all,

on my FreeBSD server

12.1-STABLE FreeBSD 12.1-STABLE r359627 P6T_WS_PRO amd64

# zpool list

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT System 2.72T 1.17T 1.55T - - 1% 42% 1.00x ONLINE - zPool 13.6T 9.17T 4.43T - - 17% 67% 1.00x ONLINE - zTest 480M 396K 480M - - 0% 0% 1.00x ONLINE - zzTOP 7.25T 4.82T 2.43T - - 0% 66% 1.00x ONLINE -

captain-haddock17 commented 4 years ago

Note : after boot and user login : sudo zpool import -a

no pools available to import

What is OK :
sudo zpool import -c /etc/zfs/zpool.cache -a zpool list

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zTest 480M 347M 133M - - 38% 72% 1.00x ONLINE -

or sudo zpool import -d /vdev 554310920175130116 is ok too.

captain-haddock17 commented 4 years ago

Log file /var/log/org.openzfsonosx/zpool-import-all.err

no pools available to import

Log file /var/log/org.openzfsonosx/zpool-import-all.log

+zpool-import-all.sh Thu Apr 9 12:43:19 CEST 2020 2020-04-09 12:43:31.352 system_profiler[578:2640] SPUSBDevice: IOCreatePlugInInterfaceForService failed 0xe00002be 12.35 real 0.85 user 0.88 sys Waiting up to 60 seconds for the InvariantDisks idle file /var/run/disk/invariant.idle to exist Found /var/run/disk/invariant.idle after 0 iterations of sleeping 0.1 seconds Thu Apr 9 12:43:32 CEST 2020 Running zpool import -a Thu Apr 9 12:43:42 CEST 2020 pool: zTest id: 554310920175130116 state: ONLINE action: The pool can be imported using its name or numeric identifier. config:

zTest ONLINE /vdev/zTest.pool ONLINE Thu Apr 9 12:43:42 CEST 2020 Finished running zpool import -a : 0 Touching the file /var/run/org.openzfsonosx.zpool-import-all.didRun Thu Apr 9 12:43:42 CEST 2020 -zpool-import-all.sh

Strange that we can see :

pool: zTest id: 554310920175130116 state: ONLINE

and zpool listafterwards says

no pool available to import

I'll try to cross-check this contradiction !

captain-haddock17 commented 4 years ago

I wonder if Launchd import.plist definition would benefit from

`

KeepAlive PathState /var/run/org.openzfsonosx.zpool-import-all.didRun `

captain-haddock17 commented 4 years ago

I wonder if Launchd of zpool-import-all.plist definition would benefit from

` KeepAlive

PathState /var/run/org.openzfsonosx.zpool-import-all.didRun

`

and why not: <key>StartOnMount</key> <true/>

import script will be fired on every disk mount, even USB (or network I guess.)

captain-haddock17 commented 4 years ago

Some Ref Doc : (you certainly have it already !)

Daemons and Services Programming Guide

cbreak-black commented 4 years ago

/var/run/disk/by-id/ is manged by InvariantDisks, it will delete all its contents when started (which happens during boot). You're not supposed to but your own links in there.

Running zpool import after a mount makes little sense, since mounting comes AFTER importing, not before. I wrote ZetaWatch, which can import pools when USB sticks are inserted or other devices appear, but this is independent from mounting, it depends on DiskArbitration notifying of the appearnce of a new device.

captain-haddock17 commented 4 years ago

Hey Jorgen,

I've got some results with an adapted /Library/LaunchDaemond/org.openzfsonosx.zpool-import-all.plist

    <key>RunAtLoad</key>
    <false/>
    <key>StartOnMount</key>
    <true/>
    <key>KeepAlive</key>
        <dict>
            <key>PathState</key>
        <dict>
            <key>/var/run/org.openzfsonosx.zpool-import-all.didRun</key>
            <false/>
        </dict>
    </dict>

and adapted import /usr/local/libexec/zfs/launchd.d/zpool-import-all.sh

export VDEV_DIR=/vdev
export ZPOOLCACHE_FILE=/etc/zfs/zpool.cache
export DISKID_DIR=/var/run/disk/by-id
(...)
echo "Running zpool import -c $ZPOOLCACHE_FILE -a"
"${ZPOOL}" import -c $ZPOOLCACHE_FILE -a
ret=$?
echo "Finished running zpool import from cache : ${ret}"
"${ZPOOL}" list

echo "Running zpool import -d $DISKID_DIR -a"
"${ZPOOL}" import -d $DISKID_DIR -a
ret=$?
echo "Finished running zpool import from disk-ID: ${ret}"
"${ZPOOL}" list

# file name must be the poolname !!
for POOL in $(ls -1 $VDEV_DIR ) 
do
    echo "Running zpool import -d $VDEV_DIR name_of_pool"
    "${ZPOOL}" import -d $VDEV_DIR $POOL
done;
ret=$?
echo "Finished running zpool import : ${ret}"
"${ZPOOL}" list

What do you think of ? Kind regards, William

openzfsonosx / zfs