Open craigsloggett opened 1 year ago
No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.
No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.
@rhatdan: Please clarify. Do you mean there are no podman devs who support zfs storage at the moment? Using overlayfs where zfs is native is not desirable.
I came to report the same bug but with a minor variation in the error:
$ STORAGE_DRIVER=zfs podman system info
Error: cannot find root filesystem rpool/var/lib/containers/storage-<user>: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/var/lib/containers/storage-<user>" => cannot open 'rpool/var/lib/containers/storage-<user>': dataset does not exist
If the error code is derived from the actual command executed, then it should be noted that: /usr/sbin/zfs fs list -rHp...
is not a valid command.
I don't know where to look but I assume someone familiar with the code could verify it isn't a basic fat-finger error with minimal effort.
zfs is not native in the upstream kernel and none of the core development team use it.
I see. How did the current level of support develop? Is the current level of support going to be deprecated?
Since this affects the basic functionality and anything ZFS related is "can't fix/won't fix" some note should be made in the docs so that for those of us who consider this a critical function won't waste our time.
This is a community project, if community wants to support it, then they need to step up. The volunteers or people who are paid to work on this stuff do not work on zfs.
Understood and agreed. Unfortunately I posess neither the required skills or bandwidth to fix.
As it stands, rootless ZFS support is non-functional. This should be documented somewhere so that users are aware. The volunteers or people who are paid to work on this stuff do have access to update the docs. It isn't a big ask to make a footnote in the appropriate location. Maybe an additonal note that devs with ZFS experience would be especially welcome...
podman is running the wrong command:
$ podman system reset -f
Error: cannot find root filesystem rpool/containers_user: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user" => cannot open 'rpool/containers_user': dataset does not exist
The correct command is: zfs list -Hp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user
This was on Debian 12's 4.3.1 though.
Interested in opening a PR?
This code comes from vendor/github.com/mistifyio/go-zfs/v3/
Interested in opening a PR?
Figuring out first how Ubuntu 24.04 makes podman work with ZFS as backing storage with overlay as driver. The bug was in Debian 12's podman 4.3.1 though.
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?
Maybe I'm wrong. Ubuntu 24.04 has podman 4.9.3 working with ZFS as backing filesystem, not as graph driver. I don't know if it uses fuse-overlayfs to get it working because trying the same without FUSE in Debian 12 fails.
FreeBSD does use podman 4.8.3 with ZFS as graph driver. Only runs as root, though.
I prefer to use VFS as graph driver over fuse-overlayfs with ZFS as backing filesystem. This is my /etc/containers/storage.conf
:
[storage]
driver = "vfs"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"
You have to remove root and user storage directories and run podman system reset -f
. You know the drill.
For reasons that are not entirely clear to me, the general consensus (between Docker and ZFS devs) is that the FUSE ZFS driver is neither performant nor entirely stable compared to the native ZOL driver. Also, VFS does not allow for the performance / storage gains of using copy-on-write, which is a core driver for using ZFS. The clone/snapshot/clone mechanism is really excellent for a PaaS environment.
What we are looking for is rootless, daemonless containers leveraging ZFS. Podman in theory does all this, but the rootless containers fail when asked to use the ZFS driver. If we run them as root they work great...but the inherent security issues remain. Since it looks like getting ZFS on rootless containers to work on Debian 12 is problematic, we are investigating moving over to Docker instead.
@ricardobranco777 Please do post back if you find a version of podman where rootless containers are working correctly with the native ZoL ZFS driver
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
Haven't tried rootless Docker with ZFS.
That is the issue, you drop Podman because zfs does not work with rootless mode, but then switch to Docker in rootful mode.
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
It appears not, which then brings us right back to podman...so we will not be migrating and will have to weigh the benefits of having rootless containers vs the performance and scaling implications of using vfs or overlayfs riding on zfs.
I'll keep an eye on this issue in hopes someone with ZFS chops takes interest and makes our dreams come true.
Issue Description
Rootless Podman is unable to list ZFS filesystems.
Here is the rootless
storage.conf
file:This is the error I get when attempting to do anything with rootless Podman:
When I run the command manually using the same user, I am able to list the filesystem just fine:
~/.local/share/containers
is owned by<user>:<user>
and here are the ZFS permissions:Steps to reproduce the issue
$ podman system reset
$ su -
# zfs create -o mountpoint=/home/<user>/.local/share/containers -o dedup=on zroot/containers
# zfs allow <user> create,destroy,mount,snapshot zroot/containers
# chown <user>:<user> /home/<user>/.local/share/containers
exit
$ podman system info
Describe the results you received
After running
podman system info
(or any Podman command as a regular user):Describe the results you expected
The
zfs
storage driver is able to "see" the dataset and returns the system info without error.podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
No response
Additional information
Interestingly, using the
zfs
storage driver asroot
works fine on the same system: