Ubuntu Trusty: pool not imported on boot/reboot

[mikedm139]

Pool is not imported or mounted on boot. I have to manually import the pool at which time it is automatically mounted.

Refreshing the zpool.cache as per "Troubleshooting" instructions yields no change.

output from: mountall --verbose kern.log fstab zpool status zfs getall dmesg

[allspark]

I've the same problem on a fresh ubuntu server installation.

[dajhorn]

I can see three things to check.

First,

The mountall log does not have the ZFS message that should appear at the --debug level. Double-check that the ubuntu-zfs package is installed and that both --verbose and --debug are set.

Second,

LSISAS2008: FWVersion(17.00.01.00), ChipRevision(0x03), BiosVersion(00.00.00.00)

A quick web search suggests that this equipment is sensitive to PSU capacity and/or firmware revision. Twenty-five disks require a large PSU, or something designed to use dual or triple sources.

If you bought these cards online for a good price to use in a home media toaster, then they're probably counterfeit or cross-flashed.

Third,

swapon: /dev/disk/by-uuid/957c71c6-8c03-4684-9585-8eb2e30937d5: swapon failed: Device or resource busy

This looks like the well-known mpt2sas problem described in the FAQ.

If the /dev/sdi5 device node is not ready when the mountall job starts, then we can expect other nodes to sporadically fail. Troubleshoot here by delaying the mountall event and/or by verifying that all /dev/disk/by-id links are pointing to a device node that actually exists at runtime.

Also note that manually partitioned pool members can aggravate this problem because slow partition scans sometimes glitch ZFS. Some people do this to remove the small slack area that a regular zpool create appends to the EFI table, and it causes a non-zero number of problems.

[mikedm139]

The mountall --verbose output in the gist linked above is from running the command as root. When running mountall --verbose as a normal user, the following is the result:

$ mountall --verbose
Failed to load ZFS module stack.
Load the module manually by running 'insmod <location>/zfs.ko' as root.
/ is local
/proc is virtual
/proc/sys/fs/binfmt_misc is virtual
/sys is virtual
/sys/fs/cgroup is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/run/user is virtual
/sys/fs/pstore is virtual
/run/rpc_pipefs is virtual
/sys/fs/cgroup/systemd is virtual
mountall: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
mounting event sent for /tmp
mountall: Connection is closed
mountall: Disconnected from Upstart

I do have the ubuntu-zfs package installed, as evidenced by:

$ dpkg-query -s ubuntu-zfs
Package: ubuntu-zfs
Status: install ok installed
Priority: optional
Section: metapackages
Installed-Size: 38
Maintainer: Darik Horn <dajhorn@vanadac.com>
Architecture: amd64
Version: 8~trusty
Depends: spl, spl-dkms, zfs-dkms, zfsutils
Recommends: build-essential, linux-headers, ubuntu-minimal, zfs-mountall
Suggests: zfs-auto-snapshot
Conflicts: libzfs1, libzpool1
Conffiles:
 /etc/apt/preferences.d/pin-zfs-native 1e3825eff09a31a8140e8a1107b522bc
Description: Native ZFS filesystem metapackage for Ubuntu.

although, I'm unsure how to confirm that --verbose and --debug are "set".

I did, in fact, get the SAS cards online and cross-flash them so that they would just act as SATA passthrough rather than RAID controllers. I'm not quite sure how to rule these out as a source of error, except that they were working fine previously (with a full drive complement). I'm running a 750W PSU which according to my calculations on http://powersupplycalculator.net/ should be ample even considering spinup surge.

The swap partition is located on the system SSD and is not a pool device. I have nevertheless disabled the swap by commenting it out in fstab but with no change in behaviour.

I have tried adding a sleep command in /etc/init/mountall.conf as suggested in the FAQ. Setting it as high as 3 minutes did not get the pool to auto-import and mount. All /dev/disk/by-id links point to existing devices.

Any further troubleshooting/debugging steps I can follow?

[dajhorn]

Failed to load ZFS module stack.
Load the module manually by running 'insmod <location>/zfs.ko' as root.

I didn't notice this error earlier, and it must be fixed first. Hopefully it isn't sporadic.

This can happen when ZoL modules are mismatched with the kernel, like after an apt-get dist-upgrade, or if the default module parameters were overridden, such as after performance tuning. Check for a coincident line in the dmesg mentioning ZoL like "SPL symbol mismatch".

Run dkms status and check that everything is in the installed state and manually loadable. If not, then follow the rebuild/reinstall instructions. If so, then reboot into a recovery prompt through the GRUB menu and check that modprobe zfs and zfs mount -a work before the regular system starts.

This can also happen when /usr, /var, or /lib are outside of the system root, but I don't see those mounts in the given /etc/fstab file. Or when the ZoL modules are accidentally put into the initrd, but I don't see that either.

I'm unsure how to confirm that --verbose and --debug are "set".

They will be on the exec mountall line in the /etc/init/mountall.conf file.

I did, in fact, get the SAS cards online and cross-flash them so that they would just act as SATA passthrough rather than RAID controllers. I'm not quite sure how to rule these out as a source of error,

I've been around long enough to notice a trend and check for it, but there isn't a good way for a small buyer to hedge for this kind of duff equipment. The iffy clones and flakey refurbs are priced well within the threshold of temptation for most people.

except that they were working fine previously (with a full drive complement).

ZFS expresses a corner case and is sensitive to races because pool members must be online together.

I'm running a 750W PSU which according to my calculations on http://powersupplycalculator.net/ should be ample even considering spinup surge.

This sticker wattage is low enough to make rail loads matter. Even if the PSU is not a contributing factor, try to configure the HBA and/or system for a staggered spin-up to get more headroom. There are nearly thirty things in this computer pulling peak on the PSU in a period of four seconds.

[mikedm139]

Thanks. To be clear, i only see that error when running as a non-root user. I'm not sure if that is important or not. I realized that my original report did not include the output of mountall --debug. I ran that as root and have attached the output here.

Both dmesg | grep ZoL and dmesg | grep symbol produce no output. dkms status returns

spl, 0.6.4.2, 3.13.0-61-generic, x86_64: installed
spl, 0.6.4.2, 3.13.0-62-generic, x86_64: installed
vboxhost, 4.3.30, 3.13.0-61-generic, x86_64: installed
vboxhost, 4.3.30, 3.13.0-62-generic, x86_64: installed
zfs, 0.6.4.2, 3.13.0-61-generic, x86_64: installed
zfs, 0.6.4.2, 3.13.0-62-generic, x86_64: installed

Running zfs manually works and asks for command arguments. Attempting to manually load spl gives an error:

$ spl
No command 'spl' found, did you mean:
 Command 'sl' from package 'sl' (universe)
 Command 'spc' from package 'supercat' (universe)
 Command 'spd' from package 'spd' (universe)
 Command 'rpl' from package 'rpl' (universe)
 Command 'sol' from package 'aisleriot' (main)
 Command 'sepl' from package 'sepia' (universe)
 Command 'sql' from package 'parallel' (universe)
 Command 'sml' from package 'smlnj' (universe)
 Command 'spe' from package 'spe' (universe)
spl: command not found

Does that mean that I need to rebuild/re-install spl?

This is the content of /etc/init/mountall.conf

# mountall - Mount filesystems on boot
#
# This helper mounts filesystems in the correct order as the devices
# and mountpoints become available.

description     "Mount filesystems on boot"

start on startup
stop on starting rcS

expect daemon
task

emits virtual-filesystems
emits local-filesystems
emits remote-filesystems
emits all-swaps
emits filesystem
emits mounting
emits mounted

script
    . /etc/default/rcS || true
    [ -f /forcefsck ] && force_fsck="--force-fsck"
    [ "$FSCKFIX" = "yes" ] && fsck_fix="--fsck-fix"

    # Doesn't work so well if mountall is responsible for mounting /proc, heh.
    if [ -e /proc/cmdline ]; then
        read line < /proc/cmdline
        for arg in $line; do
            case $arg in
                -q|--quiet|-v|--verbose|--debug)
                    debug_arg=$arg
                    ;;
            esac
        done < /proc/cmdline
    fi
    # set $LANG so that messages appearing in plymouth are translated
    if [ -r /etc/default/locale ]; then
        . /etc/default/locale || true
        export LANG LANGUAGE LC_MESSAGES LC_ALL
    fi

    exec mountall --daemon $force_fsck $fsck_fix $debug_arg
end script

post-stop script
    rm -f /forcefsck 2>dev/null || true
end script

If I'm reading it correctly, the --verbose and --debug options are eligible options to be passed to the mountall script but not set by default.

I'll look into setting the HBAs for staggered spinup, seems like a good idea and hopefully won't have a huge impact on usability for me.

[dajhorn]

I ran that as root and have attached the output here.

These lines

parse_filesystems: zfs (nodev)
parse_zfs_list: parsing ZFS list
update_mount: Media: /Media Media zfs zfsutil
spawn: mount -f -t zfs -o zfsutil Media /Media
spawn: mount Media [16727]
mount Media [16727] exited normally

mean that the failure is happening before the mountall job.

All disks in the pool must be online and ready before mountall loads the ZFS driver. If you can substitute a different HBA, then you can isolate for the mpt2sas bug.

Past that, creating a small test pool using the onboard SATA ports could be diagnostic.

Does that mean that I need to rebuild/re-install spl?

No, the modules are installed, but note that ZFS and its utilities must always run as the root user.

If I'm reading it correctly, the --verbose and --debug options are eligible options to be passed to the mountall script but not set by default.

Yes, right.

[mikedm139]

I tried to isolate each of the 3 HBAs by disconnecting them from the drives one at a time and connecting drives to the onboard SATA ports. There aren't enough onboard ports to connect all 8 drives from the disconnected HBA so I connected the 5 that I had room for leaving one drive from each VDEV disconnected (they're all raidz2). I realize that this is an imperfect test but I don't have a spare HBA to test with so, it's the best I could do. Sadly, there was no change in the failure to import/mount on boot.

As a follow-up, I created a single disk pool connected directly to the onboard port. Upon reboot, that pool also failed to import/mount. I'm not sure at this point if the main pool Media is causing the failure which then prevented the test pool from mounting or if the problem lies elsewhere. I am unsure how to safely isolate my test case without destroying my primary storage pool. Any further suggestions?

[dajhorn]

The next step would be setting init breakpoints and sampling the system state just prior to the mountall job. We need to know what is in the /dev tree, and whether it is actually working when certain important upstart events are emitted.

Pursuing this bug, unfortunately, now requires much more work. (And mpt2sas is thusly blurbed in the FAQ.)