booting from a pool on an external USB drive

[RJVB]

Originally reported here: https://github.com/zfsonlinux/grub/issues/11

I have a little issue with the root pool I have on an external USB drive. AFAIR, I could boot off it without issues, but for some reason it's started not being able to mount the pool. This is despite the fact that the pool does exist and imports fine manually. Looking at the screen output, it is almost as if the USB device is in the process of being mounted itself when the zpool import command is executed. Which is surprising as the bootloader and the /boot partition (btrfs) are on that same device.

Has anyone seen something similar, and is there something I could do about it, like adding a delay at the appropriate place?

[FransUrbo]

As a quick hack, to get it working for YOU, I'd say to modify the zfs start script within and then recreate the initrd manually.

Something like this:

cd /tmp
mkdir initrd
cd initrd
zcat /boot/<your current initrd> | cpio -i
vi scripts/zfs (see below)
find | cpio -H newc -o | gzip > /boot/<new initrd name>
vi /boot/grub/grub.cfg (see below)

Since I know you run the stable version of ZoL from the ZoL Debian GNU/Linux Wheezy repo, your scripts/zfs will look like this (the part we're interested in):

mountroot()
{
        pre_mountroot

        [ "$quiet" != "y" ] && log_begin_msg "Running /scripts/local-premount"
        run_scripts /scripts/local-premount
        [ "$quiet" != "y" ] && log_end_msg

        # Wait for all of the /dev/{hd,sd}[a-z] device nodes to appear.
        wait_for_udev

        # Load the module now to get consistent automatic pool import behavior.
        modprobe zfs

If anyone running Dailies and have the same problem, the script will (at this moment, version 0.6.3-0.9_g540ce4_wheezy - might change) look like this:

mountroot()
{
        pre_mountroot

        if type run_scripts > /dev/null 2>&1 && [ -f "/scripts/local-premount" ]
        then
                [ "$quiet" != "y" ] && $log_begin_msg "Running /scripts/local-premount"
                run_scripts /scripts/local-premount
                [ "$quiet" != "y" ] && $log_end_msg
        fi

        # Wait for all of the /dev/{hd,sd}[a-z] device nodes to appear.
        if type wait_for_udev > /dev/null 2>&1 ; then
                wait_for_udev
        elif type wait_for_dev > /dev/null 2>&1 ; then
                wait_for_dev
        fi

        # zpool import refuse to import without a valid mtab
        [ ! -f /proc/mounts ] && mount proc /proc
        [ ! -f /etc/mtab ] && cat /proc/mounts > /etc/mtab

        # Load the module, without importing any pools - we want manual
        # control over that part!
        modprobe zfs zfs_autoimport_disable=1

So what you need to do is add a sleep 30 (for starters - if that doesn't work, just increase it in 30 second increments it until it works satisfactory) between the wait_for_udev and modprobe zfs lines (that goes for both versions of the script).

For the second see below, you need to find the grub menu entry that boots your system and look for the initrd line. This is what my looks like:

        [...]
        linux   /vmlinuz-3.2.0-4-amd64 boot=zfs bootfs=rpool/ROOT/debian-1 ro quiet
        initrd  /initrd.img-3.2.0-4-amd64

Just change the /initrd.img....... to something like /initrd.img.......-MODIFIED (depending on what name you used in the gzip part).

!! WARNING !!! This is an ugly hack, and will be overwritten every time someone/something recreates the initrd. !! WARNING !!!

NOTE: Even though the initrd may not be overwritten (and won't if you specify a very special name), the grub config for that menu entry will be overwritten...

The reason I can't/don't want to do this in the version I'm distributing, is that this would add a delay for all those that don't want or need it. And it's difficult to know WHY a pool haven't been imported (did it fail because it's simply isn't available, because the admin/user specified the wrong pool name on the grub boot command line or because it's "already in use", etc, etc). I'm already trying three times to import the pool, and adding a lot of logic to try to catch this corner case in a clean, nice way that won't break it for someone else is not something I know today how to fix.

Let's leave this issue open, even if my suggestion solves your problem and hope that someone else could give me some hints on how to fix this in a clean way.

[RJVB]

Since I know you run the stable version of ZoL from the ZoL Debian GNU/Linux Wheezy repo, your scripts/zfs will look like this (the part we're interested in):

OK, will try that to see if indeed a delay is the only thing required.

!! WARNING !!! This is an ugly hack, and will be overwritten every time someone/something recreates the initrd. !! WARNING !!!

NOTE: Even though the initrd may not be overwritten (and won't if you specify a very special name), the grub config for that menu entry will be overwritten...

Doesn't grub generate entries for each initrd it finds? Or is it based on the vmlinux files present in /boot?

The reason I can't/don't want to do this in the version I'm distributing, is that this would add a delay for all those that don't

That's evident. WHat you might be able to do, though, is conceive of a way to add configurable delay that's 0 by default and depends on a config file that can be absent. I'm guessing the scripts/zfs file comes from somewhere in the initramfs package.

want or need it. And it's difficult to know WHY a pool haven't been imported (did it fail because it's simply isn't available, because the admin/user specified the wrong pool name on the grub boot command line or because it's "already in use", etc, etc). I'm

Yes, this is a problem with zpool!

already trying three times to import the pool, and adding a lot of logic to try to catch this corner case in a clean, nice way that won't break it for someone else is not something I know today how to fix.

Let's leave this issue open, even if my suggestion solves your problem and hope that someone else could give me some hints on how to fix this in a clean way.

How about calling zpool import in a loop with timeout, and continue either after a pool is found or the timeout triggered? A loop over something like

IMPORTS=`zpool import`
if [ $? != 0 ] ;then
        sleep 2
        IMPORT=""
fi

where IMPORT can act as an end condition; I'm too lazy to refigure out how to do loop counting right now.

More elegant would be to have zpool hook into whatever mechanism the kernel (or initram) provides to know if a device is currently being attached, and enter a wait state when it detects that this is indeed the case. But I don't know if that's even possible...

[FransUrbo]

Doesn't grub generate entries for each initrd it finds?

Yes, that's what I said. is conceive of a way to add configurable delay that's 0 by default and depends on a config file that can be absent.

Might be a plan...

[RJVB]

BTW, your suggestion worked. Except that I applied it directly to /usr/share/initramfs-tools/scripts/zfs O:-) 5 seconds were enough.

[dajhorn]

This issue is a variant of zfsonlinux/zfs#330. ZoL needs some udev plumbing for dynamic import and possibly a hook into zeventd from zfsonlinux/zfs#2.

[FransUrbo]

Putting ZED on the initrd might be a little to much.

Btw, I'm fixing this in my daily like so:

        modprobe zfs zfs_autoimport_disable=1

        if [ "$ZFS_INITRD_POST_MODPROBE_SLEEP" -gt '0' ]
        then
                [ "$quiet" != "y" ] && $log_begin_msg "Sleeping for $ZFS_INITRD_POST_MODPROBE_SLEEP seconds..."
                sleep "$ZFS_INITRD_POST_MODPROBE_SLEEP"
                [ "$quiet" != "y" ] && $log_end_msg
        fi

and then in /etc/default/zfs:

# Wait for this many seconds in the initrd mountroot?
# This delays startup and should be '0' on most systems.
# This might help on systems which have their ZFS root on
# a USB disk that takes just a little longer to be available
ZFS_INITRD_POST_MODPROBE_SLEEP='0'

[FransUrbo]

I'm closing this as stale.