search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.66k stars 1.76k forks source link

CENTOS 7 dataset disappears #2600

Closed dmaziuk closed 10 years ago

dmaziuk commented 10 years ago

possibly related to #2563

zfs pool worked for a while, at some point in the last few days -- maybe the last kernel update? -- something broke. There is no zfs after boot.

Modules are loaded, systemd logs no errors, nfsd starts and shares the mountpoint. Zfs pool isn't there. That happens with or without /etc/default/zfs.

smartctl is happy with the drives, "zfs import -f tank" works fine, subsequent srub reports no errors (there's no data there yet though).

From clean reboot:

[root@manta ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md127 397G 2.0G 375G 1% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 8.6M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup [root@manta ~]# lsmod | grep zfs zfs 1202168 3 zunicode 331251 1 zfs zavl 15010 1 zfs zcommon 51321 1 zfs znvpair 93262 2 zfs,zcommon spl 290139 5 zfs,zavl,zunicode,zcommon,znvpair [root@manta ~]# zpool list no pools available [root@manta ~]# zfs list no datasets available [root@manta ~]# zpool status -v no pools available [root@manta ~]# zpool import tank cannot import 'tank': pool may be in use from other system use '-f' to import anyway [root@manta ~]# zpool import -f tank [root@manta ~]# zpool status -v pool: tank state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Thu Aug 14 12:33:30 2014 config:

    NAME                                                  STATE     READ WRITE CKSUM
    tank                                                  ONLINE       0     0     0
      mirror-0                                            ONLINE       0     0     0
        ata-ST3000DM001-1CH166_W1F20LXR                   ONLINE       0     0     0
        ata-ST3000DM001-1E6166_W1F460D1                   ONLINE       0     0     0
    logs
      mirror-1                                            ONLINE       0     0     0
        ata-WDC_WD5003ABYX-01WERA1_WD-WMAYP3954112-part3  ONLINE       0     0     0
        ata-WDC_WD5003ABYX-01WERA1_WD-WMAYP4658203-part3  ONLINE       0     0     0

errors: No known data errors

DeHackEd commented 10 years ago

The commands you demonstrated above show you going from a not-imported pool to an imported pool. At the end do you still have nothing mounted?

Like LVM you need to have the pool loaded before the filesystems within become available. Also note that even with auto-import (/etc/zfs/zpool.cache exists with your pool in it) ZFS doesn't automount the filesystems contained within. It is necessary for something (you or an init script) to either import from scratch or execute zfs mount -a to get things all up.

Possibly helpful command: zfs list -t filesystem -o name,mounted,mountpoint

dmaziuk commented 10 years ago

What I'm saying is I had the pool tank set up and working and mounted at boot until I rebooted that host today. Now it is not mounted at boot anymore.

Yes, after I manually import the pool it's there. But until today it was there after reboot, I dodn't have to manually import it.

The pool disappears at reboot.

dmaziuk commented 10 years ago

This might be useful info: [root@manta ~]# rpm -q -a | grep zfs zfs-dkms-0.6.3-1.el7.centos.noarch libzfs2-0.6.3-1.el7.centos.x86_64 zfs-0.6.3-1.el7.centos.x86_64 zfs-release-1-2.el7.centos.noarch

dmaziuk commented 10 years ago

OK, now I got something: after exporting and re-importing the pool

Aug 14 14:06:31 manta zpool: cannot import 'tank': pool may be in use from other system Aug 14 14:06:31 manta zpool: use '-f' to import anyway Aug 14 14:06:31 manta systemd: zfs-import-cache.service: main process exited, code=exited, status=1/FAILURE Aug 14 14:06:31 manta systemd: Failed to start Import ZFS pools by cache file. Aug 14 14:06:31 manta systemd: Unit zfs-import-cache.service entered failed state. Aug 14 14:06:31 manta systemd: Starting Mount ZFS filesystems... Aug 14 14:06:31 manta systemd: Started Mount ZFS filesystems.

So the question is, should boot-time import run with -f, or is "may be in use from other system" a bug?

cointer commented 10 years ago

I am also having this issue on CentOS 7. I have to manually import the pool after reboot. This is an odd behavior, and as mentioned, the -f option must be passed when re-importing the pool.

Perhaps there is something happening where the pool is not being exported when the zfs service stops when reboot happens?

dmaziuk commented 10 years ago

I think the question is why zfs thinks the pool "may be in use". Is there lock of some sort that doesn't get cleared at shutdown?

I added -f to systemd's zfs-share target (or whatever it's called) for now.

cointer commented 10 years ago

It was my understanding that when the ZFS service stops, it is supposed to export all pools. The same "may be in use" behavior happens when you don't export a pool, move disks to another machine, and try to import it on that other machine.

I haven't tried it yet, but I bet if you export the pool before rebooting, it will import cleanly when the system comes back up. This obviously is not a solution, but perhaps it can be a clue as to what is happening. I can't try this on my systems at the moment, can you give it a try?

dmaziuk commented 10 years ago

You've any idea where in that fpo-systemd-s the "stop" action should go? Anyway, it doesn't look like you can actually do it: import runs with -c /etc/zfs/zpool.cache, but export doesn't like "-a", or "all", or anything other than the name of the pool. Which isn't known to the rpm maintainer.

/usr/lib/systemd/system has zfs-import-cache.service is the one where I added "-f" to zfs import, btw.

dmaziuk commented 10 years ago

Also, looking at stop() in /etc/init.d/zfs on centos 6, it only does sync. There's a commented out "zfs umount -a" -- understandable, you can't umont / at thas point -- but the thing is, on centos 6.5 it works fine without zpool export.

x042 commented 10 years ago

I am also having this same issue on CentOS 7 with latest kernel update.

I've figured out a workaround that solves the issue for me. I just made a shutdown "script" (not really much of a script, see below) which sleeps for 10 seconds and then runs zfs unmount -a. 

#!/bin/bash -

/usr/bin/sleep 10
/usr/sbin/zfs umount -a

Seems ZFS volumes aren't gracefully unmounted at shutdown/reboot? Looks like this, https://github.com/zfsonlinux/zfs/issues/2575, and https://github.com/zfsonlinux/zfs/issues/2563 might all be related.

l1k commented 10 years ago

Possibly fixed by #2766 if Dracut is used.

dmaziuk commented 10 years ago

Dunno what you mean by "if Dracut is used" -- what's used is what the plain "basic server" centos 7 install has, -- but after youm up'ing zfs/spl to 0.6.3-1.1.el7.centos and dkms to 2.2.0.3-26.el7, my /tank is gone again after reboot:

systemctl status zfs-import-cache zfs-import-cache.service - Import ZFS pools by cache file Loaded: loaded (/usr/lib/systemd/system/zfs-import-cache.service; static) Active: failed (Result: exit-code) since Fri 2014-10-10 17:27:17 EDT; 26s ago Process: 548 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)

Main PID: 548 (code=exited, status=1/FAILURE)

Oct 10 17:27:17 batfish.bmrb.wisc.edu zpool[548]: cannot import 'tank': pool may be in use from other system Oct 10 17:27:17 batfish.bmrb.wisc.edu zpool[548]: use '-f' to import anyway Oct 10 17:27:17 batfish.bmrb.wisc.edu systemd[1]: zfs-import-cache.service: main process exited, code=exited, status=1/FAILURE Oct 10 17:27:17 batfish.bmrb.wisc.edu systemd[1]: Failed to start Import ZFS pools by cache file. Oct 10 17:27:17 batfish.bmrb.wisc.edu systemd[1]: Unit zfs-import-cache.service entered failed state.

dmaziuk commented 10 years ago

PS could you please not overwrite my /usr/lib/systemd/system/zfs-import-cache.service (it has "-f") until you've actually fixed this?

l1k commented 10 years ago

@dmaziuk: "pool may be in use from other system" means the pool was not exported on shutdown. Are you using ZFS for / (root partition)? I gathered from what you wrote above that you may be using ZFS for the root partition. If you do, you need to apply the patch in #2766, this will cause a proper export of a ZFS root on shutdown. Dracut is included in CentOS since version 6, it's the framework for generating the init ramdisk.

dmaziuk commented 10 years ago

PPS this is not a ZFS root if by "root" you mean "/"

l1k commented 10 years ago

Okay. Then I was mistaken and this is NOT fixed by #2766, rather there must be an issue with the initscript or systemd unit responsible for exporting all zpools on shutdown. Which one is used, an initscript or a systemd unit?

dmaziuk commented 10 years ago

it's systemd and if you look up at the comments made on Aug 18 and 26 you'll find that yes, we know it's umount/export that's missing and it seemed to work without it on centos 6.

l1k commented 10 years ago

@dmaziuk: Please add this line to the [Unit] section in /usr/lib/systemd/system/zfs-mount.service:

Conflicts=final.target

Add this line to the [Service] section:

ExecStop=/bin/sh -c '/sbin/zpool list -H -o name | /usr/bin/xargs /sbin/zpool export'

Add this line to the [Service] section in /usr/lib/systemd/system/zfs-share.service:

ExecStop=/sbin/zfs unshare -a

Remove the -f you added in /usr/lib/systemd/system/zfs-import-cache.service so that zpools are not force-imported. Activate these changes:

# systemctl daemon-reload

Does this solve the issue?

dmaziuk commented 10 years ago

No. :(

It's weird:

So far the only thing that cosistently works is the "use '-f' to import anyway",

dmaziuk commented 10 years ago

On 2nd thoght I guess 'systemctl restart zfs-mount' does the explicit 'stop' so it's not weird that /tank is there if I reboot imediately afterwards -- it's the same as doing 'systemctl stop zfs-mount' before reboot.

Looks like on its own this systemd POS is not doing the 'stop' on reboot.

l1k commented 10 years ago

Is this with the modifications I described above? Before you applied these modifications, if you invoked systemctl stop zfs-mount before reboot, was /tank mounted on the next boot?

I'm not sure what the default log target is on CentOS, but assuming that it's the systemd journal, you may be able to see what's going on by inspecting the journal with journalctl. If this is of no use, the brute force method to get information is this: Edit /etc/systemd/system.conf and in the [Manager] section, configure:

DefaultStandardOutput=journal+console
DefaultStandardError=journal+console

Then add a sleep to the ExecStop command in /usr/lib/systemd/system/zfs-mount.service like this:

ExecStop=/bin/sh -c '/sbin/zpool list -H -o name | /usr/bin/xargs /sbin/zpool export ; /bin/sleep 120'

This gives you 2 minutes to inspect the output of the zpool commands immediately before shutdown. If the machine does not pause for 2 minutes, the ExecStop command probably isn't run at all. In that case, try changing the Conflicts clause to run the ExecStop command earlier:

Conflicts=shutdown.target

I don't use CentOS myself, thus can't debug this, but maybe we can track it down together.

dmaziuk commented 10 years ago

I'm pretty sure it's not running "zfs-mount stop" with "Conflicts=shutdown.target" either. Unfortunately I hosed my test system while fscking around with systemd logging, I'm taking this week off, and I've more important stuff to finish before I leave tonight. So I can roll you a centos 7 kvm image next week when I come back and upload it someplace.

dmaziuk commented 10 years ago

OK, got a fresh batch of round tuits and did a yum update to the latest systemd. In all cases /tank was mounted before reboot:

So ExecStop isn't run with Conflicts=shutdown.target either.

dmaziuk commented 10 years ago

PS. root@batfish system]# rpm -q systemd systemd-208-11.el7_0.4.x86_64

dmaziuk commented 10 years ago

Update: curiouser and curiouser: I just set up another centos 7 box, all zfs datasets got mounted on reboot (on the first one at least). I'll see if I can figure out what's different...

dmaziuk commented 10 years ago

Good news, sort of: after yum updating the kernel, the other machine has lost /tank too (same error: "may be in use by other system, use -f"). So at least this is reproducible: it all works until you update the kernel. (dkms rebuilding the modules is changing some magic that invalidates the cache file?)

I'll see if I get time to play with "import-scan" tomorrow.

dswartz commented 10 years ago

Wasn't there a recent change relating to hostid ?

dmaziuk notifications@github.com wrote:

Good news, sort of: after yum updating the kernel, the other machine has lost /tank too (same error: "may be in use by other system, use -f"). So at least this is reproducible: it all works until you update the kernel. (dkms rebuilding the modules is changing some magic that invalidates the cache file?)

I'll see if I get time to play with "import-scan" tomorrow.

— Reply to this email directly or view it on GitHub.

behlendorf commented 10 years ago

There was a hostid fix merged to master recently. However, it's not part of the stable repository yet. You could try updating to the testing repository and see if it resolves the issue.

dmaziuk commented 10 years ago

Looks like rpms from testing cured the 2nd machine. I'll see when I can update the 1st one and make sure it works there too.

dmaziuk commented 10 years ago

Testing rpms fix the other machine, too, so I'm calling the fix reproducible and the problem solved & closed. Thanks guys.

behlendorf commented 10 years ago

@dmaziuk Thanks for the confirming this was fixed!

rbrunner2 commented 9 years ago

Sorry for the stupid question ... I am a newbie to centos and zfs. I am running stock centos 7 and am about to allow system upgrades and then will build my first zfs system. What must I do to pick up this change? Thanks!

dmaziuk commented 9 years ago 
yum in epel-release
yum in kernel-devel zlib-devel libuuid-devel libblkid-devel libselinux-devel gcc dkms
yum in http://archive.zfsonlinux.org/epel/zfs-release.el7.noarch.rpm
yum in zfs

https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/

dmaziuk commented 9 years ago

DUH. Actually,

yum --enablerepo=zfs-testing in zfs

-- -133_g9635861 is the one with hostid fix.

rbrunner2 commented 9 years ago

Thank you so much!