Automount fails when accessing snapshots for root filesystem

[janlam7]

System information

Type	Version/Name
Distribution Name	Gentoo
Linux Kernel	4.19.72-gentoo
Architecture	x86_64
ZFS Version	0.8.2
SPL Version	0.8.2

# zfs get all bto/system/rootfs 
NAME               PROPERTY                      VALUE                         SOURCE
bto/system/rootfs  type                          filesystem                    -
bto/system/rootfs  creation                      Fri Jan 12  9:44 2018         -
bto/system/rootfs  used                          17.1G                         -
bto/system/rootfs  available                     5.33G                         -
bto/system/rootfs  referenced                    4.68G                         -
bto/system/rootfs  compressratio                 2.04x                         -
bto/system/rootfs  mounted                       yes                           -
bto/system/rootfs  quota                         none                          default
bto/system/rootfs  reservation                   none                          default
bto/system/rootfs  recordsize                    1M                            inherited from bto
bto/system/rootfs  mountpoint                    /                             local
bto/system/rootfs  sharenfs                      off                           default
bto/system/rootfs  checksum                      sha512                        inherited from bto
bto/system/rootfs  compression                   lz4                           inherited from bto
bto/system/rootfs  atime                         off                           inherited from bto
bto/system/rootfs  devices                       on                            default
bto/system/rootfs  exec                          on                            default
bto/system/rootfs  setuid                        on                            default
bto/system/rootfs  readonly                      off                           default
bto/system/rootfs  zoned                         off                           default
bto/system/rootfs  snapdir                       hidden                        default
bto/system/rootfs  aclinherit                    restricted                    default
bto/system/rootfs  createtxg                     47                            -
bto/system/rootfs  canmount                      on                            default
bto/system/rootfs  xattr                         sa                            inherited from bto
bto/system/rootfs  copies                        1                             default
bto/system/rootfs  version                       5                             -
bto/system/rootfs  utf8only                      on                            -
bto/system/rootfs  normalization                 formD                         -
bto/system/rootfs  casesensitivity               sensitive                     -
bto/system/rootfs  vscan                         off                           default
bto/system/rootfs  nbmand                        off                           default
bto/system/rootfs  sharesmb                      off                           default
bto/system/rootfs  refquota                      none                          default
bto/system/rootfs  refreservation                none                          default
bto/system/rootfs  guid                          10744756411378796574          -
bto/system/rootfs  primarycache                  all                           default
bto/system/rootfs  secondarycache                none                          inherited from bto
bto/system/rootfs  usedbysnapshots               12.5G                         -
bto/system/rootfs  usedbydataset                 4.68G                         -
bto/system/rootfs  usedbychildren                0B                            -
bto/system/rootfs  usedbyrefreservation          0B                            -
bto/system/rootfs  logbias                       latency                       default
bto/system/rootfs  objsetid                      394                           -
bto/system/rootfs  dedup                         off                           default
bto/system/rootfs  mlslabel                      none                          default
bto/system/rootfs  sync                          standard                      default
bto/system/rootfs  dnodesize                     auto                          inherited from bto
bto/system/rootfs  refcompressratio              2.09x                         -
bto/system/rootfs  written                       764M                          -
bto/system/rootfs  logicalused                   34.2G                         -
bto/system/rootfs  logicalreferenced             9.68G                         -
bto/system/rootfs  volmode                       default                       default
bto/system/rootfs  filesystem_limit              none                          default
bto/system/rootfs  snapshot_limit                none                          default
bto/system/rootfs  filesystem_count              none                          default
bto/system/rootfs  snapshot_count                none                          default
bto/system/rootfs  snapdev                       hidden                        default
bto/system/rootfs  acltype                       off                           default
bto/system/rootfs  context                       none                          default
bto/system/rootfs  fscontext                     none                          default
bto/system/rootfs  defcontext                    none                          default
bto/system/rootfs  rootcontext                   none                          default
bto/system/rootfs  relatime                      on                            temporary
bto/system/rootfs  redundant_metadata            all                           default
bto/system/rootfs  overlay                       off                           default
bto/system/rootfs  encryption                    off                           default
bto/system/rootfs  keylocation                   none                          default
bto/system/rootfs  keyformat                     none                          default
bto/system/rootfs  pbkdf2iters                   0                             default
bto/system/rootfs  special_small_blocks          0                             default

Describe the problem you're observing

After upgrading from zfs-0.8.1 to zfs-0.8.2 the snapshots under the root filesystem do not automount anymore. I have my root on zfs. Snapshots below other filesystems automount normally. The snapshots can be mounted manually, so I have a workaround.

Describe how to reproduce the problem

• Ensure your root filesystem is on zfs.
• Upgrade to zfs and zfs-kmod-0.8.2 and reboot.
• access any snapshot under /.zfs/snapshot/ and confirm that it's empty.

Include any warning/errors/backtraces from the system logs

I bisected it to commit 093bb6446120c50a7109ed7e7a0f2e76730b3160 in which case the following warning is logged in dmesg

 WARNING: Unable to automount (null)/.zfs/snapshot/20190101/bto/system/rootfs@20190101: 512

[behlendorf]

@janlam7 thank you for reporting this. Would it be possible for you to verify this small patch resolves the issue? Unfortunately, I don't have a test system handy which uses ZFS for the root filesystem.

diff --git a/module/os/linux/zfs/zfs_ctldir.c b/module/os/linux/zfs/zfs_ctldir.c
index 1e61ef0..3b2a6eb 100644
--- a/module/os/linux/zfs/zfs_ctldir.c
+++ b/module/os/linux/zfs/zfs_ctldir.c
@@ -1053,7 +1053,8 @@ zfsctl_snapshot_mount(struct path *path, int flags)
         * on mount.zfs(8).
         */
        snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
-           zfsvfs->z_vfs->vfs_mntpoint, dname(dentry));
+           zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
+           dname(dentry));

        /*
         * Multiple concurrent automounts of a snapshot are never allowed.

[janlam7]

@behlendorf I ended up applying the patch below, and indeed it solves the issue, nice! I changed the paths to match the 0.8-release branch.

diff --git a/module/zfs/zfs_ctldir.c b/module/zfs/zfs_ctldir.c
index 8acbbb61c..db583a647 100644
--- a/module/zfs/zfs_ctldir.c
+++ b/module/zfs/zfs_ctldir.c
@@ -1053,7 +1053,8 @@ zfsctl_snapshot_mount(struct path *path, int flags)
         * on mount.zfs(8).
         */
        snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
-           zfsvfs->z_vfs->vfs_mntpoint, dname(dentry));
+           zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
+            dname(dentry));

        /*
         * Multiple concurrent automounts of a snapshot are never allowed.

[jhyeon]

The above patch does not seem to be enough for me. Even with it applied to zfs-0.8.2, snapshots under /.zfs/snapshot/ are still empty. The debug message from /proc/spl/kstat/zfs/dbgmsg is different from what is quoted above in that the path begins with /sysroot/ rather than (null)/. The same debug message appears with and without the patch after accessing a snapshot under /.zfs/snapshot/. I am using linux kernel 5.2.20. I did not have this problem with zfs-0.8.1.

timestamp    message 
1571898386   spa.c:5623:spa_tryimport(): spa_tryimport: importing tank
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): LOADING
1571898386   vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/nvme0n1p3': best uberblock found for spa $import. txg 77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): LOADED
1571898386   spa_misc.c:408:spa_load_note(): spa_load($import, config trusted): UNLOADING
1571898386   spa.c:5475:spa_import(): spa_import: importing tank
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config trusted): LOADING
1571898386   vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/nvme0n1p3': best uberblock found for spa tank. txg 77982
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=77982
1571898386   mmp.c:249:mmp_thread_start(): MMP thread started pool 'tank' gethrtime 3559400002
1571898386   spa.c:7577:spa_async_request(): spa=tank async request task=1
1571898386   spa_misc.c:408:spa_load_note(): spa_load(tank, config trusted): LOADED
1571898386   spa_history.c:319:spa_history_log_sync(): txg 77984 open pool version 5000; software version unknown; uts (none) 5.2.20-1 #1 SMP PREEMPT Thu Oct 10 00:15:30 KST 2019 x86_64
1571898386   spa.c:7577:spa_async_request(): spa=tank async request task=32
1571898386   spa_history.c:319:spa_history_log_sync(): txg 77986 import pool version 5000; software version unknown; uts (none) 5.2.20-1 #1 SMP PREEMPT Thu Oct 10 00:15:30 KST 2019 x86_64
1571898386   spa_history.c:306:spa_history_log_sync(): command: zpool import -N tank
1571898424   zfs_ctldir.c:1086:zfsctl_snapshot_mount(): Unable to automount /sysroot/.zfs/snapshot/zfs-auto-snap_daily-2019-10-22-0530 error=512

[jhyeon]

I use initramfs created by dracut and openrc on the Funtoo distribution.

[hyegeek]

This issue got me as well. I too am using an initrd (though a homegrown one and not dracut).

I can mount a snapshot with the following

mount -t zfs pool/root@snap /mnt

but the snapshot files under /.zfs/snapshots are all empty.

[Ryushin]

Running Devuan Ascii. Patch worked fine against Debian's 0.8.2-3.

[jhyeon]

@Ryushin do you use initramfs?

[Ryushin]

@Ryushin do you use initramfs?

Yes, I use Sysvinit and initramfs. I also tested on a Debian Buster system with sysvinit and initramfs as well. So it worked on both Devuan and Debian.

[Ryushin]

Seems I spoke too soon. Looks like I found a different kind of bug related to this. I've applied the patch listed above. My rpool looks like this: rpool 1.27T 2.95T 176K none rpool/ROOT 145G 2.95T 50.1G / rpool/mysql 53.0G 2.95T 23.0G /var/lib/mysql rpool/mysql-log 5.79G 2.95T 1.44G /var/lib/mysql-log rpool/plexmediaserver 206G 2.95T 171G /var/lib/plexmediaserver rpool/spool 115G 2.95T 35.5G /var/spool rpool/virtual_machines 778G 2.95T 743G /var/lib/libvirt

I can see inside snapshots for rpool/ROOT: windwalker:~# ls /.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/ bin boot.tar.xz dead.letter etc lib lost+found mnt opt proc run share sys tmp var boot cdrom dev home lib64 media netshares path root sbin srv tftpboot usr

Looking inside snapshots for any other dataset in rpool results in "Too many levels of symbolic links" error: ls /var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/ ls: cannot access '/var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/': Too many levels of symbolic links