Rootless Podman with `zfs` Storage Driver -- "Error: cannot find root filesystem"

[craigsloggett]

Issue Description

Rootless Podman is unable to list ZFS filesystems.

Here is the rootless storage.conf file:

[storage]
driver = "zfs"

[storage.options.zfs]
fsname = "zroot/containers"
mountopt = "nodev"

This is the error I get when attempting to do anything with rootless Podman:

$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist

When I run the command manually using the same user, I am able to list the filesystem just fine:

$ /usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers
zroot/containers        -       290816  480957955968    /home/<user>/.local/share/containers  lz4     filesystem      -       0       290816  290816  137728  290816

~/.local/share/containers is owned by <user>:<user> and here are the ZFS permissions:

$ zfs allow zroot/containers
---- Permissions on zroot/containers ---------------------------------
Local+Descendent permissions:
        user <user> create,destroy,mount,snapshot

Steps to reproduce the issue

1. $ podman system reset
2. $ su -
3. # zfs create -o mountpoint=/home/<user>/.local/share/containers -o dedup=on zroot/containers
4. # zfs allow <user> create,destroy,mount,snapshot zroot/containers
5. # chown <user>:<user> /home/<user>/.local/share/containers
6. exit
7. $ podman system info

Describe the results you received

After running podman system info (or any Podman command as a regular user):

$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist

Describe the results you expected

The zfs storage driver is able to "see" the dataset and returns the system info without error.

podman info output

# podman --version
podman version 4.7.1

# uname -a
Linux saturn 6.5.8-arch1-1 containers/podman#1 SMP PREEMPT_DYNAMIC Thu, 19 Oct 2023 22:52:14 +0000 x86_64 GNU/Linux

# cat /etc/os-release 
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

Interestingly, using the zfs storage driver as root works fine on the same system:

# podman system info                                                                                                                                                                                                                                           [63/4503]
host:                   
  arch: amd64 
  buildahVersion: 1.32.0            
  cgroupControllers:                                             
  - cpuset     
  - cpu                        
  - io                                                
  - memory           
  - hugetlb                      
  - pids               
  - rdma     
  - misc      
  cgroupManager: systemd                         
  cgroupVersion: v2
  conmon:            
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0
  cpus: 40 
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048                         
  hostname: saturn
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.5.8-arch1-1
  linkmode: dynamic   
  logDriver: journald
  memFree: 132329963520                 
  memTotal: 135066349568          
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:                                    
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.8.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.8.0 
    package: /usr/lib/podman/netavark is owned by netavark 1.8.0-1
    path: /usr/lib/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.10-1
    path: /usr/bin/crun
    version: |-                                  
      crun version 1.10
      commit: c053c83c57551bca13ead8600237341818975974
      rundir: /run/crun
      spec: 1.0.0                    
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux          
  pasta:   
    executable: ""   
    package: "" 
    version: ""
  remoteSocket:  
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 1h 57m 42.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: zfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 480958218240
  graphRootUsed: 262144
  graphStatus:
    Compression: lz4
    Parent Dataset: zroot/var/lib/containers
    Parent Quota: "no"
    Space Available: "480957980544"
    Space Used By Parent: "241664"
    Zpool: zroot
    Zpool Health: ONLINE
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.1
  Built: 1696583554
  BuiltTime: Fri Oct  6 05:12:34 2023
  GitCommit: ef83eeb9c7482826672f3efa12db3d61c88df6c4-dirty
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.1

[rhatdan]

No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.

[f1d094]

No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.

@rhatdan: Please clarify. Do you mean there are no podman devs who support zfs storage at the moment? Using overlayfs where zfs is native is not desirable.

I came to report the same bug but with a minor variation in the error:

$ STORAGE_DRIVER=zfs podman system info Error: cannot find root filesystem rpool/var/lib/containers/storage-<user>: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/var/lib/containers/storage-<user>" => cannot open 'rpool/var/lib/containers/storage-<user>': dataset does not exist

If the error code is derived from the actual command executed, then it should be noted that: /usr/sbin/zfs fs list -rHp... is not a valid command.

I don't know where to look but I assume someone familiar with the code could verify it isn't a basic fat-finger error with minimal effort.

[rhatdan]

zfs is not native in the upstream kernel and none of the core development team use it.

[f1d094]

I see. How did the current level of support develop? Is the current level of support going to be deprecated?

[f1d094]

Since this affects the basic functionality and anything ZFS related is "can't fix/won't fix" some note should be made in the docs so that for those of us who consider this a critical function won't waste our time.

[rhatdan]

This is a community project, if community wants to support it, then they need to step up. The volunteers or people who are paid to work on this stuff do not work on zfs.

[f1d094]

Understood and agreed. Unfortunately I posess neither the required skills or bandwidth to fix.

As it stands, rootless ZFS support is non-functional. This should be documented somewhere so that users are aware. The volunteers or people who are paid to work on this stuff do have access to update the docs. It isn't a big ask to make a footnote in the appropriate location. Maybe an additonal note that devs with ZFS experience would be especially welcome...

[ricardobranco777]

podman is running the wrong command:

$ podman system reset -f
Error: cannot find root filesystem rpool/containers_user: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user" => cannot open 'rpool/containers_user': dataset does not exist

The correct command is: zfs list -Hp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user

This was on Debian 12's 4.3.1 though.

[rhatdan]

Interested in opening a PR?

[rhatdan]

This code comes from vendor/github.com/mistifyio/go-zfs/v3/

[ricardobranco777]

Interested in opening a PR?

Figuring out first how Ubuntu 24.04 makes podman work with ZFS as backing storage with overlay as driver. The bug was in Debian 12's podman 4.3.1 though.

[ricardobranco777]

I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.

[f1d094]

I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.

This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?

[ricardobranco777]

I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.

This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?

Maybe I'm wrong. Ubuntu 24.04 has podman 4.9.3 working with ZFS as backing filesystem, not as graph driver. I don't know if it uses fuse-overlayfs to get it working because trying the same without FUSE in Debian 12 fails.

FreeBSD does use podman 4.8.3 with ZFS as graph driver. Only runs as root, though.

I prefer to use VFS as graph driver over fuse-overlayfs with ZFS as backing filesystem. This is my /etc/containers/storage.conf:

[storage]
driver = "vfs"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"

You have to remove root and user storage directories and run podman system reset -f. You know the drill.

[f1d094]

For reasons that are not entirely clear to me, the general consensus (between Docker and ZFS devs) is that the FUSE ZFS driver is neither performant nor entirely stable compared to the native ZOL driver. Also, VFS does not allow for the performance / storage gains of using copy-on-write, which is a core driver for using ZFS. The clone/snapshot/clone mechanism is really excellent for a PaaS environment.

What we are looking for is rootless, daemonless containers leveraging ZFS. Podman in theory does all this, but the rootless containers fail when asked to use the ZFS driver. If we run them as root they work great...but the inherent security issues remain. Since it looks like getting ZFS on rootless containers to work on Debian 12 is problematic, we are investigating moving over to Docker instead.

@ricardobranco777 Please do post back if you find a version of podman where rootless containers are working correctly with the native ZoL ZFS driver

[rhatdan]

Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?

[ricardobranco777]

Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?

Haven't tried rootless Docker with ZFS.

[rhatdan]

That is the issue, you drop Podman because zfs does not work with rootless mode, but then switch to Docker in rootful mode.

[f1d094]

Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?

It appears not, which then brings us right back to podman...so we will not be migrating and will have to weigh the benefits of having rootless containers vs the performance and scaling implications of using vfs or overlayfs riding on zfs.

I'll keep an eye on this issue in hopes someone with ZFS chops takes interest and makes our dreams come true.