search

zfsonlinux / pkg-zfs

Native ZFS packaging for Debian and Ubuntu
https://launchpad.net/~zfs-native/+archive/daily
308 stars 55 forks source link

ZFS pool not mounted after booting #55

Closed tigloo closed 12 years ago

tigloo commented 12 years ago

I have a fresh installation of Ubuntu Precise with a ZFS root pool. Booting works without any errors. A second data pool in a 4x3TB RAIDZ2 configuration attached to an IBM 1015 (in LSI 9211-IT mode) fails to be mounted after booting the system.

Attempting to import the pool only works after using -f. Running mountall also only mounts the pool after it has been imported once with -f. I followed the FAQ and tried recreating zpool.cache as well as inserting a delay into mountall.conf, but it did not change anything. I also tried to remove the root directory where the filesystems are going to be mounted to ensure that is empty. Looking at the logs, the ZFS module is always initialized before all hard drives come online.

dmesg: https://www.dropbox.com/s/qiqu93i8wkth96i/dmesg.txt fstab: https://www.dropbox.com/s/t45cnhkqvtj0ljp/fstab.txt kern.log: https://www.dropbox.com/s/6t13p17dds32lr5/kern.txt

Note that in these logs the timer is a bit short. I experimented with the values a bit. However, even when there is a 60 second delay or more there is no change. This is the initialization with a bigger timer:

Nov  5 00:12:30 tricky kernel: [   23.108018] mpt2sas0: port enable: SUCCESS
Nov  5 00:12:30 tricky kernel: [   23.108380] sd 6:0:0:0: Attached scsi generic sg2 type 0
Nov  5 00:12:30 tricky kernel: [   23.108638] sd 6:0:1:0: Attached scsi generic sg3 type 0
Nov  5 00:12:30 tricky kernel: [   23.108881] sd 6:0:0:0: [sdc] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
Nov  5 00:12:30 tricky kernel: [   23.108894] sd 6:0:2:0: Attached scsi generic sg4 type 0
Nov  5 00:12:30 tricky kernel: [   23.108964] sd 6:0:0:0: [sdc] 4096-byte physical blocks
Nov  5 00:12:30 tricky kernel: [   23.109119] sd 6:0:1:0: [sdd] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
Nov  5 00:12:30 tricky kernel: [   23.109169] sd 6:0:1:0: [sdd] 4096-byte physical blocks
Nov  5 00:12:30 tricky kernel: [   23.109249] sd 6:0:3:0: Attached scsi generic sg5 type 0
Nov  5 00:12:30 tricky kernel: [   23.109389] sd 6:0:2:0: [sde] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
Nov  5 00:12:30 tricky kernel: [   23.109433] sd 6:0:2:0: [sde] 4096-byte physical blocks
Nov  5 00:12:30 tricky kernel: [   23.109670] sd 6:0:3:0: [sdf] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
Nov  5 00:12:30 tricky kernel: [   23.109712] sd 6:0:3:0: [sdf] 4096-byte physical blocks
Nov  5 00:12:30 tricky kernel: [   23.169336] sd 6:0:1:0: [sdd] Write Protect is off
Nov  5 00:12:30 tricky kernel: [   23.169373] sd 6:0:1:0: [sdd] Mode Sense: 7f 00 10 08
Nov  5 00:12:30 tricky kernel: [   23.170022] sd 6:0:1:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov  5 00:12:30 tricky kernel: [   23.170969] sd 6:0:3:0: [sdf] Write Protect is off
Nov  5 00:12:30 tricky kernel: [   23.171006] sd 6:0:3:0: [sdf] Mode Sense: 7f 00 10 08
Nov  5 00:12:30 tricky kernel: [   23.171292] sd 6:0:0:0: [sdc] Write Protect is off
Nov  5 00:12:30 tricky kernel: [   23.171330] sd 6:0:0:0: [sdc] Mode Sense: 7f 00 10 08
Nov  5 00:12:30 tricky kernel: [   23.171651] sd 6:0:3:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov  5 00:12:30 tricky kernel: [   23.172021] sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov  5 00:12:30 tricky kernel: [   23.174094] sd 6:0:2:0: [sde] Write Protect is off
Nov  5 00:12:30 tricky kernel: [   23.174132] sd 6:0:2:0: [sde] Mode Sense: 7f 00 10 08
Nov  5 00:12:30 tricky kernel: [   23.174784] sd 6:0:2:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov  5 00:12:30 tricky kernel: [   23.260885]  sdc: sdc1 sdc9
Nov  5 00:12:30 tricky kernel: [   23.264316]  sdf: sdf1 sdf9
Nov  5 00:12:30 tricky kernel: [   23.266718]  sde: sde1 sde9
Nov  5 00:12:30 tricky kernel: [   23.267059]  sdd: sdd1 sdd9
Nov  5 00:12:30 tricky kernel: [   23.321891] sd 6:0:0:0: [sdc] Attached SCSI disk
Nov  5 00:12:30 tricky kernel: [   23.328264] sd 6:0:1:0: [sdd] Attached SCSI disk
Nov  5 00:12:30 tricky kernel: [   23.329889] sd 6:0:3:0: [sdf] Attached SCSI disk
Nov  5 00:12:30 tricky kernel: [   23.333028] sd 6:0:2:0: [sde] Attached SCSI disk
Nov  5 00:12:30 tricky kernel: [   46.215860] Adding 16777212k swap on /dev/zd0.  Priority:-1 extents:1 across:16777212k SS
Nov  5 00:12:30 tricky kernel: [   46.587203] ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov  5 00:12:31 tricky kernel: [   48.712024] MCE: In-kernel MCE decoding enabled.

zd0 is my swap device, so mountall runs at this time. If you need the output of mountall --verbose, I can provide it via mail.

Here is the ZFS status:

till@tricky:/$ sudo zpool status
[sudo] password for till: 
  pool: rpool
 state: ONLINE
 scan: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        rpool                                      ONLINE       0     0     0
          scsi-SATA_SAMSUNG_HD501LJS0MUJ1EQ159603  ONLINE       0     0     0

errors: No known data errors
till@tricky:/$ sudo zpool import -d /dev/disk/by-id tank
cannot import 'tank': pool may be in use from other system
use '-f' to import anyway
till@tricky:/$ sudo zpool import -f -d /dev/disk/by-id tank
till@tricky:/$ sudo zpool status
  pool: rpool
 state: ONLINE
 scan: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        rpool                                      ONLINE       0     0     0
          scsi-SATA_SAMSUNG_HD501LJS0MUJ1EQ159603  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
 scan: none requested
config:

        NAME                                    STATE     READ WRITE CKSUM
        tank                                    ONLINE       0     0     0
          raidz2-0                              ONLINE       0     0     0
            scsi-SATA_ST3000DM001-9YN_S1F0MYWK  ONLINE       0     0     0
            scsi-SATA_ST3000DM001-9YN_W1F0VW8Z  ONLINE       0     0     0
            scsi-SATA_ST3000DM001-9YN_Z1F0Q831  ONLINE       0     0     0
            scsi-SATA_ST3000DM001-9YN_Z1F0QYAT  ONLINE       0     0     0

errors: No known data errors
till@tricky:/$
dajhorn commented 12 years ago

Thanks for posting the files mentioned in the FAQ. The basic problem here is that the M1015 is bringing drives online after the ZFS driver is loaded. (The upstream ticket is zfsonlinux/zfs#330.)

On this computer, the ZoL driver is in the initrd because the system has a ZFS root. This means that the mpt2sas driver must also be in the initrd. Do this:

  1. Open /etc/initramfs-tools/modules file in a text editor.
  2. Add an mpt2sas line and save the file.
  3. Run update-initramfs -c -k all.
  4. Reboot and try again.

Afterwards, in the kernel log, check that all lines like this:

Happen before this line:

tigloo commented 12 years ago

The mpt2sas driver now loads before the ZFS module, but it will still initialize the drives after the ZFS module is loaded. Is there a way I can delay loading the ZFS module?

The symptoms after booting are now as follows:

Is there a reason why mountall succeeds when it is run manually after boot, but fails when run in the init sequence? Looking at the logs, there is a 10 second delay between all drives being online and available and mountall being run.

Updated dmesg: https://www.dropbox.com/s/nxr7bjc2hm66nv0/dmesg2.txt

tigloo commented 12 years ago

Calling mountall from rc.local works around the problem.

dajhorn commented 12 years ago

The mpt2sas driver now loads before the ZFS module, but it will still initialize the drives after the ZFS module is loaded. Is there a way I can delay loading the ZFS module?

Put a sleep 60 line above the wait_for_udev line in the /usr/share/initramfs-tools/scripts file, run update-initramfs -c -k all, and reboot.

Is there a reason why mountall succeeds when it is run manually after boot, but fails when run in the init sequence?

ZoL doesn't handle hotplug events, per zfsonlinux/zfs#330, and it interacts poorly with many mpt2sas implementations. The IBM M1015 itself could also be dud hardware; it appears in many bug reports like dajhorn/pkg-zfs#53.

tigloo commented 12 years ago

Thanks, that fixed it.