Automount fails when booting up

After booting no ZFS filesystems are mounted.

System is a clean Ubuntu 12.04.1 x64, 10xWD Red 3TB in RaidZ2 of which 8 disks on an IBM M1015 HBA controller card in IT mode.

The mountall FAQ did not solve the issue. (https://github.com/dajhorn/pkg-zfs/wiki/Ubuntu-ZFS-mountall-FAQ-and-troubleshooting)

Full issue description + logs here (halfway down the page): https://github.com/zfsonlinux/zfs/issues/769

(sorry, but I'm just finding my way around these issue trackers and was sent here. If needed i'll repost the full description and logs here)

Thank you.

i tried the following:

zfs unmount tank make sure /media/data (my root mount point for my tank/ zpool) is empty; it was empty as expected delete /media/data folder zpool export tank make sure /etc/zfs/zpool.cache does not exist; it did not exist zpool import tank reboot

after the reboot my filesystems were mounted after a subsequent boot they still were

I never wrote anything to /media/data with the filesystem unmounted. It appeared empty and hidden directories and filed should have been visible. Could there be a bug where zfs thinks the directory is not empty or could a hidden .zfs folder be written to it while unmounted? (though I assume the .zfs folder is virtual and never actually on disk)

Never mind the above; on subsequent cold boot the next morning, the automounting failed again and there are no ZFS mount points.

Any comments?

I read zfsonlinux/zfs#1068 and it looks like you did everything right per the FAQ. (Thanks.)

Are you running Samba on this media share? This could be caused by something like dajhorn/pkg-zfs#43.

Could there be a bug where zfs thinks the directory is not empty or could a hidden .zfs folder be written to it while unmounted? (though I assume the .zfs folder is virtual and never actually on disk)

Yes, there is a latent bug where the kernel automounter doesn't always release the /.zfs directory in time, but in this case it could be caused by a daemon keeping an open handle on a mount point at shutdown.

Please try this:

Manually clear the mount points again.
Before reboot or shutdown, manually stop services like Samba that are running on /media/data.
Check whether the mount failure persists at system start when you do this.

If running rm /media/data temporarily resolves the problem, then put it in the mountall.conf file before the exec line to kludge the problem. The logs suggest that the sleep statement might not be necessary with version 14 firmware on the LSI controller.

Are you running Samba on this media share? This could be caused by something like dajhorn/pkg-zfs#43.

I have samba server (smbd) running as part of this server install, but I have not edited the config yet and did not make any shares. My ZFS subfilesystems only contain data at this time, and the root filesystem (tank/=/media/data) is empty.

If running rm /media/data temporarily resolves the problem, then put it in the mountall.conf file before the exec line to kludge the problem. The logs suggest that the sleep statement might not be necessary with version 14 firmware on the LSI controller.

I said 4 posts earlier that after deleting the mount point directory (/media/data) the filesystems were automounted after a reboot (2 times). Sadly today this does not work anymore. I believe it's something of chance or timing. In any case it's not consistent. (My controller card's bios version is MPT2BIOS 7.27.00.00 (2012.07.02), recent, and I do believe Rev.14 is correct)

Please try this:

Manually clear the mount points again. Before reboot or shutdown, manually stop services like Samba that are running on /media/data. Check whether the mount failure persists at system start when you do this.

In any case, I ran your test: (clean boot, then login)

root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
root@server2:~# dir /media
total 28
drwxr-xr-x  7 root root 4096 Oct 25 17:05 .
drwxr-xr-x 23 root root 4096 Oct 21 19:21 ..
drwxr-xr-x  2 root root 4096 Oct 16 18:32 cdrom
drwxr-xr-x  2 root root 4096 Oct 25 17:05 data
drwxr-xr-x  3 root root 4096 Oct 16 18:29 news
drwxr-xr-x  2 root root 4096 Oct 16 18:29 tv
drwxr-xr-x  2 root root 4096 Oct 25 03:42 usb
root@server2:~# dir /media/data
total 8
drwxr-xr-x 2 root root 4096 Oct 25 17:05 .
drwxr-xr-x 7 root root 4096 Oct 25 17:05 ..
root@server2:~# rm -r /media/data
root@server2:~# dir /media
total 24
drwxr-xr-x  6 root root 4096 Oct 25 20:20 .
drwxr-xr-x 23 root root 4096 Oct 21 19:21 ..
drwxr-xr-x  2 root root 4096 Oct 16 18:32 cdrom
drwxr-xr-x  3 root root 4096 Oct 16 18:29 news
drwxr-xr-x  2 root root 4096 Oct 16 18:29 tv
drwxr-xr-x  2 root root 4096 Oct 25 03:42 usb
root@server2:~# dir /etc/zfs/
total 16
drwxr-xr-x   2 root root 4096 Oct 25 17:05 .
drwxr-xr-x 104 root root 4096 Oct 25 20:14 ..
-rw-r--r--   1 root root  183 Sep 18 21:33 zdev.conf
-rw-r--r--   1 root root 3748 Oct 25 17:05 zpool.cache
root@server2:~# service smbd stop
smbd stop/waiting
root@server2:~# service smbd status
smbd stop/waiting
root@server2:~# shutdown -r now
Broadcast message from xxx@server2
        (/dev/pts/0) at 20:26 ...
The system is going down for reboot NOW!

post-boot:

root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
root@server2:~# dir /media
total 24
drwxr-xr-x  6 root root 4096 Oct 25 20:20 .
drwxr-xr-x 23 root root 4096 Oct 21 19:21 ..
drwxr-xr-x  2 root root 4096 Oct 16 18:32 cdrom
drwxr-xr-x  3 root root 4096 Oct 16 18:29 news
drwxr-xr-x  2 root root 4096 Oct 16 18:29 tv
drwxr-xr-x  2 root root 4096 Oct 25 03:42 usb
root@server2:~# mountall --debug
plymouth_connect: Failed to connect to Plymouth: Connection refused
parse_filesystems: reading filesystems
parse_filesystems: sysfs (nodev)
parse_filesystems: rootfs (nodev)
parse_filesystems: bdev (nodev)
parse_filesystems: proc (nodev)
parse_filesystems: cgroup (nodev)
parse_filesystems: cpuset (nodev)
parse_filesystems: tmpfs (nodev)
parse_filesystems: devtmpfs (nodev)
parse_filesystems: debugfs (nodev)
parse_filesystems: securityfs (nodev)
parse_filesystems: sockfs (nodev)
parse_filesystems: pipefs (nodev)
parse_filesystems: anon_inodefs (nodev)
parse_filesystems: devpts (nodev)
parse_filesystems: ext3
parse_filesystems: ext4
parse_filesystems: ramfs (nodev)
parse_filesystems: hugetlbfs (nodev)
parse_filesystems: ecryptfs (nodev)
parse_filesystems: fuseblk
parse_filesystems: fuse (nodev)
parse_filesystems: fusectl (nodev)
parse_filesystems: pstore (nodev)
parse_filesystems: mqueue (nodev)
parse_filesystems: zfs (nodev)
parse_fstab: updating from /lib/init/fstab
update_mount: /: / /dev/root rootfs defaults check
update_mount: /proc: /proc none proc nodev,noexec,nosuid
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc none binfmt_misc nodev,noexec,nosuid,optional
update_mount: /sys: /sys none sysfs nodev,noexec,nosuid
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /spu: /spu none spufs gid=spu,optional
update_mount: /dev: /dev none devtmpfs,tmpfs mode=0755
update_mount: /dev/pts: /dev/pts none devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /tmp: /tmp none none defaults
update_mount: /run: /run none tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
parse_fstab: updating from /etc/fstab
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /: / UUID=d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /media/news: /media/news UUID=d4ff0407-d605-40b4-b88a-a5c2c6ca3dfa ext4 noatime check
parse_zfs_list: parsing ZFS list
update_mount: /media/data: /media/data tank zfs zfsutil
update_mount: /media/data/archive: /media/data/archive tank/archive zfs zfsutil
update_mount: /media/data/audio: /media/data/audio tank/audio zfs zfsutil
update_mount: /media/data/backup: /media/data/backup tank/backup zfs zfsutil
update_mount: /media/data/dev: /media/data/dev tank/dev zfs zfsutil
update_mount: /media/data/documents: /media/data/documents tank/documents zfs zfsutil
update_mount: /media/data/ftp: /media/data/ftp tank/ftp zfs zfsutil
update_mount: /media/data/pics: /media/data/pics tank/pics zfs zfsutil
update_mount: /media/data/video: /media/data/video tank/video zfs zfsutil
update_mount: /media/data/web: /media/data/web tank/web zfs zfsutil
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /media/news: /media/news /dev/sda3 ext4 noatime check
mount_policy: /proc/sys/fs/binfmt_misc: dropping unknown filesystem
mount_policy: /spu: dropping unknown filesystem
tag_mount: / is root filesystem
mount_policy: /proc can be mounted while root readonly
mount_policy: /sys can be mounted while root readonly
mount_policy: /sys/fs/fuse/connections parent is /sys
mount_policy: /sys/kernel/debug parent is /sys
mount_policy: /sys/kernel/security parent is /sys
mount_policy: /dev can be mounted while root readonly
mount_policy: /dev/pts parent is /dev
mount_policy: /tmp parent is /
tag_mount: /tmp is not virtual, inherited local
mount_policy: /run can be mounted while root readonly
mount_policy: /run/lock parent is /run
mount_policy: /run/shm parent is /run
mount_policy: /media/news parent is /
mount_policy: /media/data can be mounted while root readonly
mount_policy: /media/data prior fstab entry /media/news
tag_mount: /media/data is not virtual, inherited local
mount_policy: /media/data/archive parent is /media/data
mount_policy: /media/data/archive prior fstab entry /media/data
tag_mount: /media/data/archive is not virtual, inherited local
mount_policy: /media/data/audio parent is /media/data
mount_policy: /media/data/audio prior fstab entry /media/data/archive
tag_mount: /media/data/audio is not virtual, inherited local
mount_policy: /media/data/backup parent is /media/data
mount_policy: /media/data/backup prior fstab entry /media/data/audio
tag_mount: /media/data/backup is not virtual, inherited local
mount_policy: /media/data/dev parent is /media/data
mount_policy: /media/data/dev prior fstab entry /media/data/backup
tag_mount: /media/data/dev is not virtual, inherited local
mount_policy: /media/data/documents parent is /media/data
mount_policy: /media/data/documents prior fstab entry /media/data/dev
tag_mount: /media/data/documents is not virtual, inherited local
mount_policy: /media/data/ftp parent is /media/data
mount_policy: /media/data/ftp prior fstab entry /media/data/documents
tag_mount: /media/data/ftp is not virtual, inherited local
mount_policy: /media/data/pics parent is /media/data
mount_policy: /media/data/pics prior fstab entry /media/data/ftp
tag_mount: /media/data/pics is not virtual, inherited local
mount_policy: /media/data/video parent is /media/data
mount_policy: /media/data/video prior fstab entry /media/data/pics
tag_mount: /media/data/video is not virtual, inherited local
mount_policy: /media/data/web parent is /media/data
mount_policy: /media/data/web prior fstab entry /media/data/video
tag_mount: /media/data/web is not virtual, inherited local
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/media/news is local
/media/data is local
/media/data/archive is local
/media/data/audio is local
/media/data/backup is local
/media/data/dev is local
/media/data/documents is local
/media/data/ftp is local
/media/data/pics is local
/media/data/video is local
/media/data/web is local
mounted: /
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_mount: mtab /
spawn: mount -f -t ext4 -o noatime,errors=remount-ro /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 /
spawn: mount / [2488]
mount / [2488] exited normally
run_mount: mtab /proc
spawn: mount -f -t proc -o nodev,noexec,nosuid proc /proc
spawn: mount /proc [2489]
mount /proc [2489] exited normally
run_mount: mtab /sys
spawn: mount -f -t sysfs -o nodev,noexec,nosuid sysfs /sys
spawn: mount /sys [2490]
mount /sys [2490] exited normally
run_mount: mtab /sys/fs/fuse/connections
spawn: mount -f -t fusectl -o  none /sys/fs/fuse/connections
spawn: mount /sys/fs/fuse/connections [2491]
mount /sys/fs/fuse/connections [2491] exited normally
run_mount: mtab /sys/kernel/debug
spawn: mount -f -t debugfs -o  none /sys/kernel/debug
spawn: mount /sys/kernel/debug [2492]
mount /sys/kernel/debug [2492] exited normally
run_mount: mtab /sys/kernel/security
spawn: mount -f -t securityfs -o  none /sys/kernel/security
spawn: mount /sys/kernel/security [2493]
mount /sys/kernel/security [2493] exited normally
run_mount: mtab /dev
spawn: mount -f -t devtmpfs -o mode=0755 udev /dev
spawn: mount /dev [2494]
mount /dev [2494] exited normally
run_mount: mtab /dev/pts
spawn: mount -f -t devpts -o noexec,nosuid,gid=tty,mode=0620 devpts /dev/pts
spawn: mount /dev/pts [2495]
mount /dev/pts [2495] exited normally
run_mount: mtab /run
spawn: mount -f -t tmpfs -o noexec,nosuid,size=10%,mode=0755 tmpfs /run
spawn: mount /run [2496]
mount /run [2496] exited normally
run_mount: mtab /run/lock
spawn: mount -f -t tmpfs -o nodev,noexec,nosuid,size=5242880 none /run/lock
spawn: mount /run/lock [2497]
mount /run/lock [2497] exited normally
run_mount: mtab /run/shm
spawn: mount -f -t tmpfs -o nosuid,nodev none /run/shm
spawn: mount /run/shm [2498]
mount /run/shm [2498] exited normally
run_mount: mtab /media/news
spawn: mount -f -t ext4 -o noatime /dev/sda3 /media/news
spawn: mount /media/news [2499]
mount /media/news [2499] exited normally
swap finished
local 1/13 remote 0/0 virtual 0/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /proc
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 1/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 2/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/fuse/connections
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 3/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/kernel/debug
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 4/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/kernel/security
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 5/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /dev
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 6/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /dev/pts
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 7/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 8/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run/lock
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 1/13 remote 0/0 virtual 9/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run/shm
plymouth_connect: Failed to connect to Plymouth: Connection refused
virtual finished
remote finished
local 1/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /media/news
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 2/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
control_server_open: Mountall0_1.Server started at address: unix:abstract=/com/ubuntu/mountall/server/
run_fsck: /tmp: no check required
run_mount: /tmp: placeholder
mounted: /tmp
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 3/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_fsck: /media/data: no check required
mounting /media/data
spawn: mount -t zfs -o zfsutil tank /media/data
spawn: mount /media/data [2547]
mount /media/data [2547] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /media/news: /media/news /dev/sda3 ext4 noatime check
update_mount: /media/data: /media/data tank zfs zfsutil
mounted: /media/data
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 4/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_fsck: /media/data/archive: no check required
mounting /media/data/archive
spawn: mount -t zfs -o zfsutil tank/archive /media/data/archive
spawn: mount /media/data/archive [2568]
mount /media/data/archive [2568] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /media/news: /media/news /dev/sda3 ext4 noatime check
update_mount: /media/data: /media/data tank zfs zfsutil
update_mount: /media/data/archive: /media/data/archive tank/archive zfs zfsutil
mounted: /media/data/archive
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 5/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_fsck: /media/data/audio: no check required
mounting /media/data/audio
spawn: mount -t zfs -o zfsutil tank/audio /media/data/audio
spawn: mount /media/data/audio [2680]
mount /media/data/audio [2680] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /media/news: /media/news /dev/sda3 ext4 noatime check
update_mount: /media/data: /media/data tank zfs zfsutil
update_mount: /media/data/archive: /media/data/archive tank/archive zfs zfsutil
update_mount: /media/data/audio: /media/data/audio tank/audio zfs zfsutil
mounted: /media/data/audio
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 6/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_fsck: /media/data/backup: no check required
mounting /media/data/backup
spawn: mount -t zfs -o zfsutil tank/backup /media/data/backup
spawn: mount /media/data/backup [2699]
mount /media/data/backup [2699] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 ext4 noatime,errors=remount-ro check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /media/news: /media/news /dev/sda3 ext4 noatime check
update_mount: /media/data: /media/data tank zfs zfsutil
update_mount: /media/data/archive: /media/data/archive tank/archive zfs zfsutil
update_mount: /media/data/audio: /media/data/audio tank/audio zfs zfsutil
update_mount: /media/data/backup: /media/data/backup tank/backup zfs zfsutil
mounted: /media/data/backup
plymouth_connect: Failed to connect to Plymouth: Connection refused
local 7/13 remote 0/0 virtual 10/10 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
run_fsck: /media/data/dev: no check required
mounting /media/data/dev
spawn: mount -t zfs -o zfsutil tank/dev /media/data/dev
spawn: mount /media/data/dev [2702]
filesystem 'tank/dev' is already mounted
mountall: mount /media/data/dev [2702] terminated with status 2
mountall: Filesystem could not be mounted: /media/data/dev
plymouth_connect: Failed to connect to Plymouth: Connection refused
mountall: Skipping mounting /media/data/dev since Plymouth is not available
skip_mount: /media/data/dev
skip_mount: /media/data/documents
skip_mount: /media/data/ftp
skip_mount: /media/data/pics
skip_mount: /media/data/video
skip_mount: /media/data/web
local finished
All filesystems mounted
local 7/7 remote 0/0 virtual 10/10 swap 0/0
local 7/7 remote 0/0 virtual 10/10 swap 0/0
local 7/7 remote 0/0 virtual 10/10 swap 0/0
local 7/7 remote 0/0 virtual 10/10 swap 0/0
local 7/7 remote 0/0 virtual 10/10 swap 0/0
local 7/7 remote 0/0 virtual 10/10 swap 0/0
udev_catchup: catching up
try_udev_device: block /dev/sdd1 6349401902125761056 tank
try_udev_device: block /dev/sde1 6349401902125761056 tank
try_udev_device: block /dev/sdf1 6349401902125761056 tank
try_udev_device: block /dev/sdg1 6349401902125761056 tank
try_udev_device: block /dev/sdh1 6349401902125761056 tank
try_udev_device: block /dev/sdi1 6349401902125761056 tank
try_udev_device: block /dev/sdj1 6349401902125761056 tank
try_udev_device: block /dev/sdk1 6349401902125761056 tank
try_udev_device: block /dev/sda1 f4397b60-35c2-c3c4-6225-f675aa035d51 server2:0
try_udev_device: block /dev/sda2 e0e15ef9-60a7-4f58-8bb7-4b310c694e79 (null)
try_udev_device: block /dev/sda3 d4ff0407-d605-40b4-b88a-a5c2c6ca3dfa (null)
try_udev_device: /media/news by name
run_fsck: /media/news: mounted filesystem
run_mount: /media/news: already mounted
try_udev_device: block /dev/sdb1 6349401902125761056 tank
try_udev_device: block /dev/sdc1 6349401902125761056 tank
try_udev_device: block /dev/md0 d6a9cd7f-3ab0-4bfa-842c-26bb64573e61 boot
try_udev_device: / by link /dev/disk/by-uuid/d6a9cd7f-3ab0-4bfa-842c-26bb64573e61
run_fsck: /: mounted filesystem
run_mount: /: already mounted

Now I am missing about half my filesystems. One half is mounted, the other is not. Weird(!) I carry on trying to mount.

(continued..)

root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank on /media/data type zfs (rw)
tank/archive on /media/data/archive type zfs (rw)
tank/audio on /media/data/audio type zfs (rw)
tank/backup on /media/data/backup type zfs (rw)
root@server2:~# mountall --verbose
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/media/news is local
/media/data/dev is local
/media/data/documents is local
/media/data/ftp is local
/media/data/pics is local
/media/data/video is local
/media/data/web is local
/media/data is local
/media/data/archive is local
/media/data/audio is local
/media/data/backup is local
mount / [2718] exited normally
mount /proc [2719] exited normally
mount /sys [2720] exited normally
mount /sys/fs/fuse/connections [2721] exited normally
mount /sys/kernel/debug [2722] exited normally
mount /sys/kernel/security [2723] exited normally
mount /dev [2724] exited normally
mount /dev/pts [2725] exited normally
mount /run [2726] exited normally
mount /run/lock [2727] exited normally
mount /run/shm [2728] exited normally
mount /media/news [2729] exited normally
filesystem 'tank' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank'.
See zfs(8) for more information.
mountall: mount /media/data [2730] terminated with status 1
filesystem 'tank/archive' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/archive'.
See zfs(8) for more information.
mountall: mount /media/data/archive [2733] terminated with status 1
filesystem 'tank/audio' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/audio'.
See zfs(8) for more information.
mountall: mount /media/data/audio [2735] terminated with status 1
filesystem 'tank/backup' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/backup'.
See zfs(8) for more information.
mountall: mount /media/data/backup [2737] terminated with status 1
swap finished
local 1/13 remote 0/0 virtual 0/10 swap 0/0
local 1/13 remote 0/0 virtual 1/10 swap 0/0
local 1/13 remote 0/0 virtual 2/10 swap 0/0
local 1/13 remote 0/0 virtual 3/10 swap 0/0
local 1/13 remote 0/0 virtual 4/10 swap 0/0
local 1/13 remote 0/0 virtual 5/10 swap 0/0
local 1/13 remote 0/0 virtual 6/10 swap 0/0
local 1/13 remote 0/0 virtual 7/10 swap 0/0
local 1/13 remote 0/0 virtual 8/10 swap 0/0
local 1/13 remote 0/0 virtual 9/10 swap 0/0
virtual finished
remote finished
local 1/13 remote 0/0 virtual 10/10 swap 0/0
local 2/13 remote 0/0 virtual 10/10 swap 0/0
local 3/13 remote 0/0 virtual 10/10 swap 0/0
local 4/13 remote 0/0 virtual 10/10 swap 0/0
local 5/13 remote 0/0 virtual 10/10 swap 0/0
local 6/13 remote 0/0 virtual 10/10 swap 0/0
local 7/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/dev
mount /media/data/dev [2786] exited normally
local 8/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/documents
mount /media/data/documents [2790] exited normally
local 9/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/ftp
mount /media/data/ftp [2820] exited normally
local 10/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/pics
mount /media/data/pics [2874] exited normally
local 11/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/video
mount /media/data/video [2924] exited normally
local 12/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/web
mount /media/data/web [2927] exited normally
local finished
All filesystems mounted
local 13/13 remote 0/0 virtual 10/10 swap 0/0
root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank/dev on /media/data/dev type zfs (rw)
tank/documents on /media/data/documents type zfs (rw)
tank/ftp on /media/data/ftp type zfs (rw)
tank/pics on /media/data/pics type zfs (rw)
tank/video on /media/data/video type zfs (rw)
tank/web on /media/data/web type zfs (rw)
root@server2:~# zpool status
  pool: tank
 state: ONLINE
 scan: none requested
config:
        NAME                                           STATE     READ WRITE CKSUM
        tank                                           ONLINE       0     0     0
          raidz2-0                                     ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0061334  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0074252  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0074906  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0075172  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0076274  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0078222  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0568807  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0580606  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0656559  ONLINE       0     0     0
            scsi-SATA_WDC_WD30EFRX-68_WD-WMC1T0658861  ONLINE       0     0     0
errors: No known data errors
root@server2:~# mountall
filesystem 'tank/dev' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/dev'.
See zfs(8) for more information.
mountall: mount /media/data/dev [2963] terminated with status 1
filesystem 'tank/documents' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/documents'.
See zfs(8) for more information.
mountall: mount /media/data/documents [2965] terminated with status 1
filesystem 'tank/ftp' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/ftp'.
See zfs(8) for more information.
mountall: mount /media/data/ftp [2967] terminated with status 1
filesystem 'tank/pics' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/pics'.
See zfs(8) for more information.
mountall: mount /media/data/pics [2969] terminated with status 1
filesystem 'tank/video' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/video'.
See zfs(8) for more information.
mountall: mount /media/data/video [2971] terminated with status 1
filesystem 'tank/web' cannot be mounted using 'mount'.
Use 'zfs set mountpoint=legacy' or 'zfs mount tank/web'.
See zfs(8) for more information.
mountall: mount /media/data/web [2973] terminated with status 1
root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank on /media/data type zfs (rw)
tank/archive on /media/data/archive type zfs (rw)
tank/audio on /media/data/audio type zfs (rw)
tank/backup on /media/data/backup type zfs (rw)

So now I get these strange errors as shown above;

mountall: mount /media/data/video [2971] terminated with status 1
filesystem 'tank/web' cannot be mounted using 'mount'.

Still only half my filesystems. Trying to export the pool

root@server2:~# zpool export tank
umount: /media/data: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
cannot unmount '/media/data': umount failed
root@server2:~# lsof | egrep -i "zfs|media"
zfs_iput_ 2480       root  cwd       DIR                9,0     4096          2 /
zfs_iput_ 2480       root  rtd       DIR                9,0     4096          2 /
zfs_iput_ 2480       root  txt   unknown                                        /proc/2480/exe

Rebooting.

(continued, after boot)

root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
root@server2:~# dir /media
total 28
drwxr-xr-x  7 root root 4096 Oct 25 20:30 .
drwxr-xr-x 23 root root 4096 Oct 21 19:21 ..
drwxr-xr-x  2 root root 4096 Oct 16 18:32 cdrom
drwxr-xr-x  2 root root 4096 Oct 25 20:30 data
drwxr-xr-x  3 root root 4096 Oct 16 18:29 news
drwxr-xr-x  2 root root 4096 Oct 16 18:29 tv
drwxr-xr-x  2 root root 4096 Oct 25 03:42 usb
root@server2:~# mountall --verbose
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/media/news is local
/media/data is local
/media/data/archive is local
/media/data/audio is local
/media/data/backup is local
/media/data/dev is local
/media/data/documents is local
/media/data/ftp is local
/media/data/pics is local
/media/data/video is local
/media/data/web is local
mount / [2501] exited normally
mount /proc [2502] exited normally
mount /sys [2503] exited normally
mount /sys/fs/fuse/connections [2504] exited normally
mount /sys/kernel/debug [2505] exited normally
mount /sys/kernel/security [2506] exited normally
mount /dev [2507] exited normally
mount /dev/pts [2508] exited normally
mount /run [2509] exited normally
mount /run/lock [2510] exited normally
mount /run/shm [2511] exited normally
mount /media/news [2512] exited normally
swap finished
local 1/13 remote 0/0 virtual 0/10 swap 0/0
local 1/13 remote 0/0 virtual 1/10 swap 0/0
local 1/13 remote 0/0 virtual 2/10 swap 0/0
local 1/13 remote 0/0 virtual 3/10 swap 0/0
local 1/13 remote 0/0 virtual 4/10 swap 0/0
local 1/13 remote 0/0 virtual 5/10 swap 0/0
local 1/13 remote 0/0 virtual 6/10 swap 0/0
local 1/13 remote 0/0 virtual 7/10 swap 0/0
local 1/13 remote 0/0 virtual 8/10 swap 0/0
local 1/13 remote 0/0 virtual 9/10 swap 0/0
virtual finished
remote finished
local 1/13 remote 0/0 virtual 10/10 swap 0/0
local 2/13 remote 0/0 virtual 10/10 swap 0/0
local 3/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data
mount /media/data [2560] exited normally
local 4/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/archive
mount /media/data/archive [2569] exited normally
local 5/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/audio
mount /media/data/audio [2632] exited normally
local 6/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/backup
mount /media/data/backup [2683] exited normally
local 7/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/dev
mount /media/data/dev [2698] exited normally
local 8/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/documents
mount /media/data/documents [2701] exited normally
local 9/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/ftp
mount /media/data/ftp [2704] exited normally
local 10/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/pics
mount /media/data/pics [2707] exited normally
local 11/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/video
mount /media/data/video [2710] exited normally
local 12/13 remote 0/0 virtual 10/10 swap 0/0
mounting /media/data/web
mount /media/data/web [2713] exited normally
local finished
All filesystems mounted
local 13/13 remote 0/0 virtual 10/10 swap 0/0
root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank on /media/data type zfs (rw)
tank/archive on /media/data/archive type zfs (rw)
tank/audio on /media/data/audio type zfs (rw)
tank/backup on /media/data/backup type zfs (rw)
tank/dev on /media/data/dev type zfs (rw)
tank/documents on /media/data/documents type zfs (rw)
tank/ftp on /media/data/ftp type zfs (rw)
tank/pics on /media/data/pics type zfs (rw)
tank/video on /media/data/video type zfs (rw)
tank/web on /media/data/web type zfs (rw)
root@server2:~#

Presto. All filesystems accounted for (but no automount).

For now, the only thing that seems to work is to put 'zfs mount -a' in rc.local, but that seems like an ugly hack and more importantly I am afraid it will cause problems later on as I install services that may require the filesystems to be available before rc.local gets executed.

What did caught my eye when using 'zfs mount -a' in the rc.local file, was that it caused a pause of about 4 seconds during boot, after the zfs drivers were loaded. I have never seen booting pause like that. I could tell it had to be the zfs mount command, because issuing a manual mountall, when successful, takes the same time. In contrast, when mountall fails, it hardly takes time.

Also a question - could adapter settings like 'IO Device Missing Delay' and 'Report Device Missing Delay' (currently 8 seconds, each) have any influence?

root@server2:~# lsof | egrep -i "zfs|media" zfsiput 2480 root cwd DIR 9,0 4096 2 / zfsiput 2480 root rtd DIR 9,0 4096 2 / zfsiput 2480 root txt unknown /proc/2480/exe

That is the zfs_iput_taskq kernel process. It seems like something is holding ZFS mount points open at the wrong time.

The /media tree is usually controlled by an automounter like the gnome-volume-manager. Try this:

# zfs umount -a
# zfs set mountpoint=/data tank
# rmdir /media/data
# zfs mount -a
# ln -s /data /media/data
# reboot

For now, the only thing that seems to work is to put 'zfs mount -a' in rc.local, but that seems like an ugly hack and more importantly I am afraid it will cause problems later on as I install services that may require the filesystems to be available before rc.local gets executed.

Right. Any service that is invoked by an upstart event will race the sysv call.

I just turned of the server for a few minutes and added a disk (Samsung HD103UJ 1TB) with some data that I want to copy to it. During booting I saw a warning or error message; something like '../dev/zfs already exist...', or doesn't exist. I only saw it a split second. I searched dmesg for it but (both manually and with grep) but it is not there. The noteworthy thing here is, however, that I had all my ZFS mounts after booting.

I did a subsequent warm reboot and all my mounts were gone again. I also tried to catch that message again but it did not show.

Try this: ...

I tried that (without the symlink as I don't want to add any more complexity to this problem, and I don't really care where it's mounted if it solves this problem);

root@server2:~# mountall
root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank on /media/data type zfs (rw)
tank/archive on /media/data/archive type zfs (rw)
tank/audio on /media/data/audio type zfs (rw)
tank/backup on /media/data/backup type zfs (rw)
tank/dev on /media/data/dev type zfs (rw)
tank/documents on /media/data/documents type zfs (rw)
tank/ftp on /media/data/ftp type zfs (rw)
tank/pics on /media/data/pics type zfs (rw)
tank/video on /media/data/video type zfs (rw)
tank/web on /media/data/web type zfs (rw)
root@server2:~# zfs umount -a
root@server2:~# zfs set mountpoint=/data tank
root@server2:~# dir /media
total 28
drwxr-xr-x  7 root root 4096 Oct 25 21:35 .
drwxr-xr-x 23 root root 4096 Oct 21 19:21 ..
drwxr-xr-x  2 root root 4096 Oct 16 18:32 cdrom
drwxr-xr-x  2 root root 4096 Oct 25 21:35 data
drwxr-xr-x  3 root root 4096 Oct 16 18:29 news
drwxr-xr-x  2 root root 4096 Oct 16 18:29 tv
drwxr-xr-x  2 root root 4096 Oct 25 03:42 usb
root@server2:~# dir /media/data
total 8
drwxr-xr-x 2 root root 4096 Oct 25 21:35 .
drwxr-xr-x 7 root root 4096 Oct 25 21:35 ..
root@server2:~# rmdir /media/data
root@server2:~# dir /media/data
ls: cannot access /media/data: No such file or directory
root@server2:~# dir /data
ls: cannot access /data: No such file or directory
root@server2:~# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank            4.46T  16.0T   475K  /data
tank/archive     141G  16.0T   141G  /data/archive
tank/audio      21.9G  16.0T  21.9G  /data/audio
tank/backup     36.1G  16.0T  36.1G  /data/backup
tank/dev        8.02G  16.0T  8.02G  /data/dev
tank/documents  8.07G  16.0T  8.07G  /data/documents
tank/ftp        37.6G  16.0T  37.6G  /data/ftp
tank/pics       35.6G  16.0T  35.6G  /data/pics
tank/video      4.17T  16.0T  4.17T  /data/video
tank/web        1.75G  16.0T  1.75G  /data/web
root@server2:~# zfs mount -a
root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
tank on /data type zfs (rw,noatime,xattr)
tank/archive on /data/archive type zfs (rw,noatime,xattr)
tank/audio on /data/audio type zfs (rw,noatime,xattr)
tank/backup on /data/backup type zfs (rw,noatime,xattr)
tank/dev on /data/dev type zfs (rw,noatime,xattr)
tank/documents on /data/documents type zfs (rw,noatime,xattr)
tank/ftp on /data/ftp type zfs (rw,noatime,xattr)
tank/pics on /data/pics type zfs (rw,noatime,xattr)
tank/video on /data/video type zfs (rw,noatime,xattr)
tank/web on /data/web type zfs (rw,noatime,xattr)
root@server2:~# reboot

root@server2:~# mount
/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda3 on /media/news type ext4 (rw,noatime)
root@server2:~# zfs list
NAME             USED  AVAIL  REFER  MOUNTPOINT
tank            4.46T  16.0T   475K  /data
tank/archive     141G  16.0T   141G  /data/archive
tank/audio      21.9G  16.0T  21.9G  /data/audio
tank/backup     36.1G  16.0T  36.1G  /data/backup
tank/dev        8.02G  16.0T  8.02G  /data/dev
tank/documents  8.07G  16.0T  8.07G  /data/documents
tank/ftp        37.6G  16.0T  37.6G  /data/ftp
tank/pics       35.6G  16.0T  35.6G  /data/pics
tank/video      4.17T  16.0T  4.17T  /data/video
tank/web        1.75G  16.0T  1.75G  /data/web
root@server2:~# dir /data
total 8
drwxr-xr-x  2 root root 4096 Oct 25 22:00 .
drwxr-xr-x 24 root root 4096 Oct 25 22:00 ..

I had good hopes for your suggestion that some ubuntu automounter may be locking files on /media, bud sadly mounting on /data gives the same behaviour :/

I just did another cold boot (about 1 minute wait), and the filesystem mounts are there again (on /data). Followed up with a warm boot ('# reboot'), twice, and the mounts are gone in both cases.

Again a cold boot (1 min), and -again- all my mounts were there. Subsequent warm boot they were gone.

You can shoot me now.

Could it be some piece of memory isn't cleared upon reset?

Update: another short cold boot (10 secs) -> no mounts another long cold boot (a few minutes) -> mounts available

Update2: I updated my BIOS. There were a few 'stability' updates. No changes however, exept that my filesystems mounted after the reset following the BIOS update. Subsequent warm reset didnt automount the filesystems, though. A following cold boot (few minutes) made the mounts available again.

I think by now it's clear that automount fails following a warm reset, and succeeds after a proper cold boot. Something is not cleared properly in some way. Note that in all these tests I did not use the hardware reset button (since my case doesn't have one). I can scramble one and plug it in if it matters, though.

Does any of this make any sense?

I think by now it's clear that automount fails following a warm reset, and succeeds after a proper cold boot.

Unfortunately, this makes it sound like a hardware fault.

Do the drives appear in the POST message after reboot or hard reset? Do they temporarily disappear from the bus?

Does any of this make any sense?

Nope. This is a unique bug report.

If you want to pursue the problem, then begin isolating system components.

Primarily:

What happens if you replace the M1015 with any other HBA?

Secondarily:

Is this an old Linux installation? Does it happen on a new Linux installation?
Does it happen on any other similar computer? Or just this one computer?
Does it happen on a non-Ubuntu (or non-Debian) system?

If I was sitting at the console, then I would break into a debugger during the mountall invocation and dump the system state. Unfortunately, this isn't something that I can reasonably walk you through.

If you have a business requirement to get this particular hardware kit working, then send me a hardware sample and I will diagnose the failure.

Yeah hardware failure occured to me too but no component has failed sofar. Before installing I've run memtest, tested each drive for a few hours, monitored temperatures, installed Ubuntu a few times, build some kernels on the server, then ran some performance tests and copied 7TB data. Nothing that gave me any hint of component failure or instability.

Do the drives appear in the POST message after reboot or hard reset? Do they temporarily disappear from the bus?

I have seen no failures in the post message. All drives are reported as they should. The 8 drives connected to the HBA also report properly no matter how i boot.

What happens if you replace the M1015 with any other HBA? If you have a business requirement..

Unfortunately I have none available at the moment (none that gives me the 8 ports i need to test). The system is for home use and as such my budget is limited.

Does it happen on any other similar computer? Or just this one computer?

I have not tested. I have a few around that are being used but I could kidnap one and with some work test it against the HBA card and the disks. There is a difference however; the new server giving this behaviour has an UEFI BIOS, the others a normal one. Does UEFI BIOS impact the ZFS software in some way?

Does it happen on a non-Ubuntu (or non-Debian) system?

That's a good idea. I could try, for example, the ZfsGuru FreeBSD boot CD (http://zfsguru.com/). I have connections with a co-developer. Does mounting the pool on BSD pose any risk for my data?

Is this an old Linux installation? Does it happen on a new Linux installation?

It's the newest standard Ubuntu 12.04.1 64 bit Server version. Installed it from the Canonical CD image which I have tested for corruption prior to installation. After installation I did a dist upgrade, then some minor tools, and then zfs. I could possibly try another version. Does ZFS work with kernel 3.6 yet? I had problems earlier and it turned out there was a bug in the installation package.

If I was sitting at the console, then I would break into a debugger during the mountall invocation and dump the system state.

Unfortunately, this isn't something that I can reasonably walk you through. I could set up an account and forward the SSH port to give you access, although that's not realy 'at the console' and the network might not be available at that point. On the other hand, I have a bsc. in electronics and IT, a fair share of programming experience, and know my way around a debugger. I'm no expert on linux, yet, but I'm willing to give it a try if you can give me some pointers.

Let me know what you think. Meanwhile I'll try a few more things. I'm not the person to give up easily. Lastly I'd like to thank you for your help and your effort trying to solve this. It's appreciated. I hope I can shoot you a message when I have a question regarding this.

There is a difference however; the new server giving this behaviour has an UEFI BIOS, the others a normal one. Does UEFI BIOS impact the ZFS software in some way?

Yes. Try toggling the CSM option (if it exists) in the UEFI configuration.

You shouldn't be getting different initialization behavior for a cold boot vice a warm boot.

Does mounting the pool on BSD pose any risk for my data?

No.

Does ZFS work with kernel 3.6 yet?

The PPA does not yet build for Linux 3.6 kernels.

Try toggling the CSM option (if it exists) in the UEFI configuration.

I'm not that lucky. No option.

Does mounting the pool on BSD pose any risk for my data? No.

So I have run that ZFSGuru live CD on the HBA card and it detects and mounts, no problem.

You shouldn't be getting different initialization behavior for a cold boot vice a warm boot.

I know, hence the "you can shoot me now" earlier. That said, I have at least 1 cold boot where they didn't mount earlier, so maybe it's not that consistent. I still have found no failing hardware. I have reset my UEFI BIOS to the factory values and removed the BIOS from the HBA card to eliminate it's possible influence (and since I don't boot from it I don't need it). I have also removed the 2 IVTV cards. It's as bare as I can make it. My server now consistently does -not- automount at boot.

I'm reverting to the idea that this is a timing issue. Just before writing this I read dmesg and noted that the mptsas driver and all 8 disks were neatly loaded before the SPL & ZFS drivers, but look at the dmesg from my reboot a minute ago (after i deinstalled snmpd):

[    2.615580] udevd[92]: starting version 175
[    2.651072] md: linear personality registered for level -1
[    2.665021] md: multipath personality registered for level -4
[    2.672807] md: raid0 personality registered for level 0
[    2.682324] md: raid1 personality registered for level 1
[    2.690134] async_tx: api initialized (async)
[    2.752079] Refined TSC clocksource calibration: 1649.982 MHz.
[    2.752141] Switching to clocksource tsc
[    2.756075] raid6: int64x1    615 MB/s
[    2.824091] raid6: int64x2    899 MB/s
[    2.892030] raid6: int64x4    789 MB/s
[    2.960049] raid6: int64x8    808 MB/s
[    3.028031] raid6: sse2x1    1060 MB/s
[    3.096030] raid6: sse2x2    1512 MB/s
[    3.164034] raid6: sse2x4    2233 MB/s
[    3.164079] raid6: using algorithm sse2x4 (2233 MB/s)
[    3.167363] xor: automatically using best checksumming function: generic_sse
[    3.177753] mpt2sas version 10.100.00.00 loaded
[    3.184011]    generic_sse:  3279.000 MB/sec
[    3.184059] xor: using function: generic_sse (3279.000 MB/sec)
[    3.185870] md: raid6 personality registered for level 6
[    3.185926] md: raid5 personality registered for level 5
[    3.185973] md: raid4 personality registered for level 4
[    3.188471] input: Microsoft Microsoft USB Wireless Mouse as /devices/pci0000:00/0000:00:13.0/usb5/5-1/5-1:1.0/input/input3
[    3.188614] scsi4 : Fusion MPT SAS Host
[    3.189082] generic-usb 0003:045E:00B9.0001: input,hidraw0: USB HID v1.11 Mouse [Microsoft Microsoft USB Wireless Mouse] on usb-0000:00:13.0-1/input0
[    3.189195] usbcore: registered new interface driver usbhid
[    3.189243] usbhid: USB HID core driver
[    3.192849] mpt2sas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    3.193139] mpt2sas 0000:01:00.0: setting latency timer to 64
[    3.193152] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16029320 kB)
[    3.193487] mpt2sas 0000:01:00.0: irq 43 for MSI/MSI-X
[    3.193527] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 43
[    3.193592] mpt2sas0: iomem(0x00000000feac0000), mapped(0xffffc900125c8000), size(16384)
[    3.193651] mpt2sas0: ioport(0x000000000000e000), size(256)
[    3.195594] md: raid10 personality registered for level 10
[    3.270266] scsi5 : pata_atiixp
[    3.272918] scsi6 : pata_atiixp
[    3.273303] ata5: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xf100 irq 14
[    3.273358] ata6: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xf108 irq 15
[    3.277842] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    3.281656] r8169 0000:04:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    3.281805] r8169 0000:04:00.0: setting latency timer to 64
[    3.282580] r8169 0000:04:00.0: irq 44 for MSI/MSI-X
[    3.283749] r8169 0000:04:00.0: eth0: RTL8168e/8111e at 0xffffc900125ce000, 54:04:a6:cc:b0:50, XID 0c200000 IRQ 44
[    3.283843] r8169 0000:04:00.0: eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[    3.455712] firewire_ohci 0000:06:02.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    3.465906] md: bind
[    3.469487] bio: create slab  at 1
[    3.469866] md/raid1:md0: active with 1 out of 2 mirrors
[    3.469974] md0: detected capacity change from 0 to 29981802496
[    3.499156]  md0: unknown partition table
[    3.520239] firewire_ohci: Added fw-ohci device 0000:06:02.0, OHCI v1.10, 4 IR + 8 IT contexts, quirks 0x11
[    3.660253] mpt2sas0: Allocated physical memory: size(7418 kB)
[    3.660317] mpt2sas0: Current Controller Queue Depth(3307), Max Controller Queue Depth(3432)
[    3.660376] mpt2sas0: Scatter Gather Elements per IO(128)
[    3.893050] mpt2sas0: LSISAS2008: FWVersion(14.00.00.00), ChipRevision(0x03), BiosVersion(00.00.00.00)
[    3.893169] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    3.893955] mpt2sas0: sending port enable !!
[    4.016963] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
[    4.020312] firewire_core: created device fw0: GUID 001e8c000054e16a, S400
[    5.474895] mpt2sas0: host_add: handle(0x0001), sas_addr(0x500605b003e715d0), phys(8)
[    5.789530] SPL: Loaded module v0.6.0.80-rc11
[    5.803229] zunicode: module license 'CDDL' taints kernel.
[    5.803300] Disabling lock debugging due to kernel taint
[    5.972062] scsi 4:0:0:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    5.972142] scsi 4:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x0000000000000000)
[    5.972205] scsi 4:0:0:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(3)
[    5.972413] scsi 4:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    5.972521] scsi 4:0:0:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    6.027052] SPL: using hostid 0x007f0101
[    6.029459] ZFS: Loaded module v0.6.0.80-rc11, ZFS pool version 28, ZFS filesystem version 5
[    6.221717] scsi 4:0:1:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    6.221813] scsi 4:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x0000000000000000)
[    6.221876] scsi 4:0:1:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(2)
[    6.222063] scsi 4:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    6.222192] scsi 4:0:1:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    6.471995] scsi 4:0:2:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    6.472126] scsi 4:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[    6.472203] scsi 4:0:2:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(0)
[    6.472411] scsi 4:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    6.472541] scsi 4:0:2:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    6.721915] scsi 4:0:3:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    6.722047] scsi 4:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221102000000), phy(2), device_name(0x0000000000000000)
[    6.722110] scsi 4:0:3:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(1)
[    6.722282] scsi 4:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    6.722367] scsi 4:0:3:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    6.826094] ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.947813] udevd[591]: starting version 175
[    6.971569] scsi 4:0:4:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    6.971588] scsi 4:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221104000000), phy(4), device_name(0x0000000000000000)
[    6.971594] scsi 4:0:4:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(7)
[    6.971700] scsi 4:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    6.971709] scsi 4:0:4:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.222625] scsi 4:0:5:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    7.222646] scsi 4:0:5:0: SATA: handle(0x000e), sas_addr(0x4433221105000000), phy(5), device_name(0x0000000000000000)
[    7.222651] scsi 4:0:5:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(6)
[    7.222807] scsi 4:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.222830] scsi 4:0:5:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.471778] scsi 4:0:6:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    7.471804] scsi 4:0:6:0: SATA: handle(0x000f), sas_addr(0x4433221106000000), phy(6), device_name(0x0000000000000000)
[    7.471810] scsi 4:0:6:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(5)
[    7.471929] scsi 4:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.471943] scsi 4:0:6:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.722668] scsi 4:0:7:0: Direct-Access     ATA      WDC WD30EFRX-68A 0A80 PQ: 0 ANSI: 6
[    7.722687] scsi 4:0:7:0: SATA: handle(0x0010), sas_addr(0x4433221107000000), phy(7), device_name(0x0000000000000000)
[    7.722692] scsi 4:0:7:0: SATA: enclosure_logical_id(0x500605b003e715d0), slot(4)
[    7.722798] scsi 4:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.722807] scsi 4:0:7:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    8.052954] wmi: Mapper loaded
[    8.804063] parport_pc 00:04: reported by Plug and Play ACPI
[    8.804158] parport0: PC-style at 0x378, irq 5 [PCSPP]
[    8.944719] piix4_smbus 0000:00:14.0: SMBus Host Controller at 0xb00, revision 0
[    9.005530] [drm] Initialized drm 1.1.0 20060810
[    9.040357] asus_wmi: ASUS WMI generic driver loaded
[    9.057123] SP5100 TCO timer: SP5100 TCO WatchDog Timer Driver v0.01
[    9.057285] SP5100 TCO timer: mmio address 0xbafe00 already in use
[    9.064488] lp0: using parport0 (interrupt-driven).
[    9.229599] snd_hda_intel 0000:00:01.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
[    9.229703] snd_hda_intel 0000:00:01.1: irq 45 for MSI/MSI-X
[    9.229743] snd_hda_intel 0000:00:01.1: setting latency timer to 64
[    9.441343] asus_wmi: Initialization: 0x0
[    9.441396] asus_wmi: BIOS WMI version: 0.9
[    9.441490] asus_wmi: SFUN value: 0x0
[    9.442889] input: Eee PC WMI hotkeys as /devices/platform/eeepc-wmi/input/input4
[    9.455699] ppdev: user-space parallel port driver
[    9.535806] HDMI status: Codec=0 Pin=3 Presence_Detect=0 ELD_Valid=0
[    9.535966] input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.1/sound/card0/input5
[    9.536664] snd_hda_intel 0000:00:14.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    9.690437] hda_codec: ALC887-VD: BIOS auto-probing.
[    9.698825] input: HDA ATI SB Line as /devices/pci0000:00/0000:00:14.2/sound/card1/input6
[    9.699271] input: HDA ATI SB Front Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input7
[    9.699510] input: HDA ATI SB Rear Mic as /devices/pci0000:00/0000:00:14.2/sound/card1/input8
[    9.699742] input: HDA ATI SB Front Headphone as /devices/pci0000:00/0000:00:14.2/sound/card1/input9
[    9.699984] input: HDA ATI SB Line-Out as /devices/pci0000:00/0000:00:14.2/sound/card1/input10
[    9.845864] [drm] radeon defaulting to kernel modesetting.
[    9.845872] [drm] radeon kernel modesetting enabled.
[    9.845959] radeon 0000:00:01.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    9.845969] radeon 0000:00:01.0: setting latency timer to 64
[    9.847083] [drm] initializing kernel modesetting (PALM 0x1002:0x9806 0x1043:0x84E7).
[    9.847197] [drm] register mmio base: 0xFEB00000
[    9.847200] [drm] register mmio size: 262144
[    9.847365] ATOM BIOS: AMD
[    9.847417] radeon 0000:00:01.0: VRAM: 384M 0x0000000000000000 - 0x0000000017FFFFFF (384M used)
[    9.847422] radeon 0000:00:01.0: GTT: 512M 0x0000000018000000 - 0x0000000037FFFFFF
[    9.847752] [drm] Detected VRAM RAM=384M, BAR=256M
[    9.847759] [drm] RAM width 32bits DDR
[    9.847931] [TTM] Zone  kernel: Available graphics memory: 8014660 kiB.
[    9.847935] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB.
[    9.847938] [TTM] Initializing pool allocator.
[    9.848074] [drm] radeon: 384M of VRAM memory ready
[    9.848078] [drm] radeon: 512M of GTT memory ready.
[    9.848106] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    9.848109] [drm] Driver supports precise vblank timestamp query.
[    9.848192] radeon 0000:00:01.0: irq 46 for MSI/MSI-X
[    9.848201] radeon 0000:00:01.0: radeon: using MSI.
[    9.848251] [drm] radeon: irq initialized.
[    9.848278] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    9.853541] [drm] Loading PALM Microcode
[   10.520450] type=1400 audit(1351376862.734:2): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=816 comm="apparmor_parser"
[   10.521100] type=1400 audit(1351376862.734:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=816 comm="apparmor_parser"
[   10.521471] type=1400 audit(1351376862.734:4): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=816 comm="apparmor_parser"
[   10.521688] type=1400 audit(1351376862.734:5): apparmor="STATUS" operation="profile_replace" name="/sbin/dhclient" pid=815 comm="apparmor_parser"
[   10.522388] type=1400 audit(1351376862.734:6): apparmor="STATUS" operation="profile_replace" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=815 comm="apparmor_parser"
[   10.522775] type=1400 audit(1351376862.734:7): apparmor="STATUS" operation="profile_replace" name="/usr/lib/connman/scripts/dhclient-script" pid=815 comm="apparmor_parser"
[   10.600150] mpt2sas0: port enable: SUCCESS
[   10.602574] sd 4:0:0:0: Attached scsi generic sg3 type 0
[   10.603182] sd 4:0:1:0: Attached scsi generic sg4 type 0
[   10.603717] sd 4:0:0:0: [sdd] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.603725] sd 4:0:0:0: [sdd] 4096-byte physical blocks
[   10.603747] sd 4:0:1:0: [sde] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.603751] sd 4:0:1:0: [sde] 4096-byte physical blocks
[   10.603761] sd 4:0:2:0: Attached scsi generic sg5 type 0
[   10.604147] sd 4:0:2:0: [sdf] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.604151] sd 4:0:2:0: [sdf] 4096-byte physical blocks
[   10.604320] sd 4:0:3:0: Attached scsi generic sg6 type 0
[   10.604729] sd 4:0:3:0: [sdg] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.604734] sd 4:0:3:0: [sdg] 4096-byte physical blocks
[   10.605195] sd 4:0:4:0: [sdh] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.605200] sd 4:0:4:0: [sdh] 4096-byte physical blocks
[   10.609077] sd 4:0:4:0: Attached scsi generic sg7 type 0
[   10.609712] sd 4:0:5:0: Attached scsi generic sg8 type 0
[   10.609748] sd 4:0:0:0: [sdd] Write Protect is off
[   10.609759] sd 4:0:0:0: [sdd] Mode Sense: 7f 00 10 08
[   10.609941] sd 4:0:2:0: [sdf] Write Protect is off
[   10.609946] sd 4:0:2:0: [sdf] Mode Sense: 7f 00 10 08
[   10.610100] sd 4:0:1:0: [sde] Write Protect is off
[   10.610105] sd 4:0:1:0: [sde] Mode Sense: 7f 00 10 08
[   10.610148] sd 4:0:3:0: [sdg] Write Protect is off
[   10.610153] sd 4:0:3:0: [sdg] Mode Sense: 7f 00 10 08
[   10.610196] sd 4:0:6:0: Attached scsi generic sg9 type 0
[   10.610442] sd 4:0:4:0: [sdh] Write Protect is off
[   10.610447] sd 4:0:4:0: [sdh] Mode Sense: 7f 00 10 08
[   10.610633] sd 4:0:0:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.610717] sd 4:0:7:0: Attached scsi generic sg10 type 0
[   10.611202] sd 4:0:3:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.611393] sd 4:0:4:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.611594] sd 4:0:2:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.611899] sd 4:0:1:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.617703] sd 4:0:5:0: [sdi] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.617710] sd 4:0:5:0: [sdi] 4096-byte physical blocks
[   10.621275] sd 4:0:6:0: [sdj] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.621282] sd 4:0:6:0: [sdj] 4096-byte physical blocks
[   10.622669] sd 4:0:5:0: [sdi] Write Protect is off
[   10.622678] sd 4:0:5:0: [sdi] Mode Sense: 7f 00 10 08
[   10.624126] sd 4:0:5:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.626532] sd 4:0:6:0: [sdj] Write Protect is off
[   10.626542] sd 4:0:6:0: [sdj] Mode Sense: 7f 00 10 08
[   10.627420] sd 4:0:6:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.631611] sd 4:0:7:0: [sdk] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
[   10.631619] sd 4:0:7:0: [sdk] 4096-byte physical blocks
[   10.637675] sd 4:0:7:0: [sdk] Write Protect is off
[   10.637686] sd 4:0:7:0: [sdk] Mode Sense: 7f 00 10 08
[   10.639561] sd 4:0:7:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   10.679450]  sdh: sdh1 sdh9
[   10.683914]  sde: sde1 sde9
[   10.684316]  sdf: sdf1 sdf9
[   10.685361]  sdg: sdg1 sdg9
[   10.687975] sd 4:0:4:0: [sdh] Attached SCSI disk
[   10.692321] sd 4:0:2:0: [sdf] Attached SCSI disk
[   10.692628] sd 4:0:1:0: [sde] Attached SCSI disk
[   10.692936] sd 4:0:3:0: [sdg] Attached SCSI disk
[   10.693657]  sdd: sdd1 sdd9
[   10.694897]  sdj: sdj1 sdj9
[   10.701511] sd 4:0:0:0: [sdd] Attached SCSI disk
[   10.702498] sd 4:0:6:0: [sdj] Attached SCSI disk
[   10.703548]  sdi: sdi1 sdi9
[   10.711470] sd 4:0:5:0: [sdi] Attached SCSI disk
[   10.717672]  sdk: sdk1 sdk9
[   10.725609] sd 4:0:7:0: [sdk] Attached SCSI disk
[   10.796108] [drm] PCIE GART of 512M enabled (table at 0x0000000000040000).
[   10.796453] radeon 0000:00:01.0: WB enabled
[   10.824784] [drm] ring test succeeded in 1 usecs
[   10.825012] [drm] radeon: ib pool ready.
[   10.825128] [drm] ib test succeeded in 0 usecs
[   10.836502] [drm] Radeon Display Connectors
[   10.836513] [drm] Connector 0:
[   10.836517] [drm]   HDMI-A
[   10.836520] [drm]   HPD1
[   10.836524] [drm]   DDC: 0x6430 0x6430 0x6434 0x6434 0x6438 0x6438 0x643c 0x643c
[   10.836527] [drm]   Encoders:
[   10.836530] [drm]     DFP1: INTERNAL_UNIPHY
[   10.836533] [drm] Connector 1:
[   10.836535] [drm]   DVI-D
[   10.836537] [drm]   HPD2
[   10.836541] [drm]   DDC: 0x6440 0x6440 0x6444 0x6444 0x6448 0x6448 0x644c 0x644c
[   10.836544] [drm]   Encoders:
[   10.836546] [drm]     DFP2: INTERNAL_UNIPHY
[   10.836548] [drm] Connector 2:
[   10.836550] [drm]   VGA
[   10.836554] [drm]   DDC: 0x64d8 0x64d8 0x64dc 0x64dc 0x64e0 0x64e0 0x64e4 0x64e4
[   10.836557] [drm]   Encoders:
[   10.836559] [drm]     CRT1: INTERNAL_KLDSCP_DAC1
[   10.836739] [drm] Internal thermal controller without fan control
[   10.839614] [drm] radeon: power management initialized
[   11.083612] [drm] fb mappable at 0xC0142000
[   11.083618] [drm] vram apper at 0xC0000000
[   11.083621] [drm] size 5242880
[   11.083624] [drm] fb depth is 24
[   11.083626] [drm]    pitch is 5120
[   11.083913] fbcon: radeondrmfb (fb0) is primary device
[   11.135464] Console: switching to colour frame buffer device 160x64
[   11.142584] fb0: radeondrmfb frame buffer device
[   11.142588] drm: registered panic notifier
[   11.142612] [drm] Initialized radeon 2.12.0 20080528 for 0000:00:01.0 on minor 0
[   11.511780] r8169 0000:04:00.0: eth0: link down
[   11.511801] r8169 0000:04:00.0: eth0: link down
[   11.515102] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   11.932412] EXT4-fs (md0): re-mounted. Opts: errors=remount-ro
[   12.948713] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
[   14.048838] r8169 0000:04:00.0: eth0: link up
[   14.052150] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   15.758736] init: smbd main process (1195) killed by HUP signal
[   15.758805] init: smbd main process ended, respawning
[   16.851304] init: failsafe main process (1177) killed by TERM signal
[   17.376696] type=1400 audit(1351376869.590:8): apparmor="STATUS" operation="profile_replace" name="/sbin/dhclient" pid=1551 comm="apparmor_parser"
[   17.377387] type=1400 audit(1351376869.590:9): apparmor="STATUS" operation="profile_replace" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=1551 comm="apparmor_parser"
[   17.377775] type=1400 audit(1351376869.590:10): apparmor="STATUS" operation="profile_replace" name="/usr/lib/connman/scripts/dhclient-script" pid=1551 comm="apparmor_parser"
[   17.414974] type=1400 audit(1351376869.626:11): apparmor="STATUS" operation="profile_load" name="/usr/sbin/tcpdump" pid=1554 comm="apparmor_parser"
[   17.442735] type=1400 audit(1351376869.654:12): apparmor="STATUS" operation="profile_load" name="/usr/sbin/mysqld" pid=1552 comm="apparmor_parser"
[   17.887764] type=1400 audit(1351376870.098:13): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=1610 comm="apparmor_parser"
[   23.167096] postgres (1717): /proc/1717/oom_adj is deprecated, please use /proc/1717/oom_score_adj instead.
[   24.976204] eth0: no IPv6 routers present

SPL & ZFS are loaded before the mptsas drives are initialised! And the difference was removing the snmpd package (that generated some errors in dmesg).

So, having spent this afternoon reading up on the upstart system (upstart cookbook), this raises a few questions with me:

1) Does the dmesg log actually have accurate timing and order since upstart is used? Is it still reliable as a view considering write delays and multithreaded job executions?

2) How does zfs mountall get executed? Does it use upstart? I figured there would be an upstart job for it that ran before filesystems but after mountall, but I could not find an upstart job for it.

SPL & ZFS are loaded before the mptsas drives are initialised! And the difference was removing the snmpd package (that generated some errors in dmesg).

Weird, but this is now a permutation of the well-known mpt2sas problem that should be kludged by putting sleep 30 above the exec line in the /etc/init/mountall.conf file.

In this case, the ZFS driver must be loaded after t=10.725609 when the last disk comes online.

1) Does the dmesg log actually have accurate timing and order since upstart is used? Is it still reliable as a view considering write delays and multithreaded job executions?

Yes, at least for this purpose. The number in the first column is a kernel timer that counts the number of seconds since initialization.

2) How does zfs mountall get executed?

On Ubuntu, the zfs mountall -a command is not executed during normal system start.

Does it use upstart? I figured there would be an upstart job for it that ran before filesystems but after mountall, but I could not find an upstart job for it.

Yes, through the /etc/init/mountall.conf file. The magic is in the patched /sbin/mountall binary that is installed by the PPA.

Weird, but this is now a permutation of the well-known mpt2sas problem that should be kludged by putting sleep 30 above the exec line in the /etc/init/mountall.conf file.

Well, as it turned out that happend only once, and after that I was back to square one: Disks get initialised and after that the ZFS drivers load. A subsequent sleep in mountall.conf didn't help, and why should it, the order seems to be right..

So, I'm still not convinced it's quite right. The mpt2sas might be slow to initialise its disks but upstart is an event and dependency driven system so it shoudlnt matter how long it takes. And still it does. Perhaps something is not right with the 'mountall' command in mountall.conf as it refuses to mount zfs at that time, or maybe the mpt3sas drivers saying they're initialised while they're not.

I figured to split loading the ZFS drivers and mounting zfs. I created a proper upstart job to mount zfs after the mountall job, which load the drivers, is done.

create this file: /etc/init/zfs-mount.conf

# zfs-mount - Mount the ZFS filesystems
#
# Atempts to (re)mount the ZFS filesystems after he mountall command has run..
# since the ZFS-on-Linux version of the mountall command does not always mount the.
# zfs filesystems on systems with a mpt2sas HBA card, it's useful to let this
# job redo that as soon as the filesystems are initialised.
description  "Mount the ZFS filesystems"
# start after all the filesystems from /etc/fstab have been initialised by the mountall job
start on filesystem
task
script
    # if zfs is not mounted at boot up, try inserting a sleep time below
    sleep 1
    echo "ZFS: (Re)mounting filesystems." > /dev/kmsg
    zfs mount -a
    echo "ZFS: (Re)mounted. Available mounts:" > /dev/kmsg
    mount | grep -i "type zfs" > /dev/kmsg
end script

This works like a charm :)

The earliest event on which I could get it to work was 'filesystem' - after the mountall job has loaded all the filesystems in fstab.

The sleep 1 is required though, because without it it sometimes automounts, sometimes not. I also have the feeling that using more disks increases the time before you can mount because when I tested with 6 disks a while back I didn't have these problems. Note that in contrast to putting a sleep in mountall.comf, this creates time -after- the drivers are loaded.

Perhaps it's an idea to verify in the mountall command that the filesystems have actually been mounted. I can imagine a retry mechanism with a fail timer which tries to automount for a number of seconds. Boot time isn't as important as having your data available after all.

I'm not sure this is strictly an mptsas issue - on both my laptop and desktop, with ZFS root's I see the exact same behavior (as well as on my home fileserver but that does have mpt2sas so it's neither here nor there).

The only filesystem getting reliably mounted is the root itself, which is marked as legacy in fstab and mounted by the zfs-initramfs script elsewhere.

Using the script above though seems to solve the problem, though it does make me worry about race-conditions (emitting local-filesystems is taking a gambit that the script finishes before any daemons that want things do - though with apt-cacher-ng it seems to work).

@wrouesnel: See dajhorn/pkg-zfs#63. Somebody found a bug that looks like this one.

Just wanting to leave a note that, after two months, the above script is still working on my operational server, without a glitch or problem at all.

@durandalTR: I'm thinking about including something like it for the R build series, perhaps enabled by a ZFS_KLUDGES option in the /etc/default/zfs file.

It's an old bug, but I suddenly see same behaviour on my server after reinstalling the OS (Ubuntu 14.04) on a SSD instead of a HD. The workaround init script works, but I am concerned about the issues with depending services that use my ZFS drive as home dir.

I tried all workarounds and recomendations from the wiki and sleep x did not do anything.

Because my whole OS loads in 3 seconds I figure this is some kind of race condition. Realizing that many more people will switch to SSD in the future I think this should be solved more properly.

This is on a fresh install of Ubuntu 14.04 x64 on a SSD with 3 HD's for ZFS RaidZ1. When booted, mountall works ok, but during boot itself it won't mount them automatically. Also no error messages anywhere can be found.

I have exact the same issue as peterdk with a fresh install of Ubuntu 14.04 on a SSD. The same configuration with Ubuntu 12.04 never gave me a problem with automounting. The only thing that works for now is to use the legacy mountpoint but I don't really like it for many reasons.

The packages for Utopic now have an enhancement that should generally resolve this kind of issue. All Ubuntu build series will get it before the ZoL 0.6.4 release.

Please open new tickets for similar issues that happen on systems that have the /etc/init/zpool-import.conf file.

All FAQ bullets for Ubuntu still apply, particularly:

Don't load the zfs module before the mountall job.
Don't put anything into the initrd.
Don't export pools at shutdown.

Best practices for other distributions do not apply to Ubuntu in this regard.

zfsonlinux / pkg-zfs

Automount fails when booting up #53