Silverblue: Podman complains kernel not supporting overlay fs (backing file system is unsupported for this graph driver)

rugk commented 3 years ago

/kind bug

Description Podman is completly broken on Fedora Silverblue. I did not do anything, but just tried to use podman. It throws me the error shown below or is just stuck and does not continue.

Steps to reproduce the issue: Run podman login or basically any other command.

Describe the results you received:

$ podman login …                   
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve

Describe the results you expected: No error.

Additional information you deem important (e.g. issue happens only occasionally): Happens always.

Output of podman version:

does not work

podman -v gives:

podman version 2.1.1

Output of podman info --debug:

impossible to run (no output)

Package info (e.g. output of rpm -q podman or apt list podman): This is Fedora Silverblue 33. ostree://fedora:fedora/33/x86_64/silverblue Version: 33.20201207.0 (2020-12-07T01:04:03Z)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

I did not change anything. I've set REGISTRY_AUTH_FILE, but that should not affect the whole system. Podman could be used a few months ago. I did not test it in the meantime, but now everything fails.

So I've obviously looked that up and found e.g. this: https://github.com/containers/podman/issues/7501 https://github.com/containers/podman/issues/7396

Nothing helped. E.g. I could delete (move) ~/.local/share/containers, but of course I don't want to loose all my container and b) it actually sometimes shows a different behavior: When I simply run a podman ps it shows me an empty terminal. It get's stuck. No output, nothing. It also does not exit though (unless I Ctrl+C it, of course.) I.e. the thing it advises me: Deleting libpod actually did not help to fix this.

Also tried configuring ~/.config/containers/storage.conf, but I don't know what to set there. Just work as usual in Silverblue. I don't know or care what storage driver you use. :D

Even toolbox fails fundamentally:

$ toolbox list                  
Error: failed to get the Podman version

Unfortunately podman info --debug is also broken. (Does not output anything/does not exist) Only podman -v works.

Reported downstream: https://bugzilla.redhat.com/show_bug.cgi?id=1907228

Edit: BTW, I can see it re-creates the $HOME/.local/share/containers dir when I start it though. Actually, only one file is created though: $HOME/.local/share/containers/storage/libpod/bolt_state.db

rugk commented 3 years ago

Also opened discussion in Fedora commmunity forum: https://discussion.fedoraproject.org/t/podman-broken-on-silverblue-podman-complains-about-overwritten-graph-driver-and-wants-libpod-to-be-deleted/25403?u=rugk

rugk commented 3 years ago

Oh after a system upgrade and reboot, that included an update of podman:

podman 2:2.1.1-12.fc33 -> 2:2.2.1-1.fc33
podman-plugins 2:2.1.1-12.fc33 -> 2:2.2.1-1.fc33

I now get a slightly different error message:

$ podman version
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay" 
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Note this still seems to be the default option.

Also funny, if I rerun this, I only get the error message (no ERRO[0000] is displayed anymore (which is useless anyway and I don't even get what this tries to tell me TBH. Looks like the strange and useless error codes you know from Windows :stuck_out_tongue_winking_eye:):

$ podman version
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

mheon commented 3 years ago

The original error message is not an error - it's a warning that the configuration file is being overwritten by the value in the database to ensure you still have access to your containers. It is not fatal, all Podman functionality should continue working. Most likely cause is a new storage.conf file specifying overlayfs instead of vfs appeared after upgrade.

Can you provide your ~/.config/containers/storage.conf (if it exists) and /etc/containers/storage.conf.

rugk commented 3 years ago

~/.config/containers/storage.conf is empty (respectively I tried things, but well...).

$ cat ~/.config/containers/storage.conf   
#[storage]
#driver = "overlay"
#[storage.options]
#mount_program = "/usr/bin/fuse-overlayfs"
$ cat /etc/containers/storage.conf                                 
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to the UIDs/GIDs as they should appear outside of the container,
# and the length of the range of UIDs/GIDs.  Additional mapped sets can be
# listed and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and then a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped in-container ID,
# until all of the entries have been used for maps.
#
# remap-user = "containers"
# remap-group = "containers"

# Root-auto-userns-user is a user name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid and /etc/subgid file.  These ranges will be partitioned
# to containers configured to create automatically a user namespace.  Containers
# configured to automatically create a user namespace can still overlap with containers
# having an explicit mapping set.
# This setting is ignored when running as rootless.
# root-auto-userns-user = "storage"
#
# Auto-userns-min-size is the minimum size for a user namespace created automatically.
# auto-userns-min-size=1024
#
# Auto-userns-max-size is the minimum size for a user namespace created automatically.
# auto-userns-max-size=65536

[storage.options.overlay]
# ignore_chown_errors can be set to allow a non privileged user running with
# a single UID within a user namespace to run containers. The user can pull
# and use any image even those with multiple uids.  Note multiple UIDs will be
# squashed down to the default uid in the container.  These images will have no
# separation between the users in the container. Only supported for the overlay
# and vfs drivers.
#ignore_chown_errors = "false"

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev,metacopy=on"

# Set to skip a PRIVATE bind mount on the storage home directory.
# skip_mount_home = "false"

# Size is used to set a maximum size of the container image.
# size = ""

# ForceMask specifies the permissions mask that is used for new files and
# directories.
#
# The values "shared" and "private" are accepted.
# Octal permission masks are also accepted.
#
#  "": No value specified.
#     All files/directories, get set with the permissions identified within the
#     image.
#  "private": it is equivalent to 0700.
#     All files/directories get set with 0700 permissions.  The owner has rwx
#     access to the files. No other users on the system can access the files.
#     This setting could be used with networked based homedirs.
#  "shared": it is equivalent to 0755.
#     The owner has rwx access to the files and everyone else can read, access
#     and execute them. This setting is useful for sharing containers storage
#     with other users.  For instance have a storage owned by root but shared
#     to rootless users as an additional store.
#     NOTE:  All files within the image are made readable and executable by any
#     user on the system. Even /etc/shadow within your image is now readable by
#     any user.
#
#   OCTAL: Users can experiment with other OCTAL Permissions.
#
#  Note: The force_mask Flag is an experimental feature, it could change in the
#  future.  When "force_mask" is set the original permission mask is stored in
#  the "user.containers.override_stat" xattr and the "mount_program" option must
#  be specified. Mount programs like "/usr/bin/fuse-overlayfs" present the
#  extended attribute permissions to processes within containers rather then the
#  "force_mask"  permissions.
#
# force_mask = ""

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

mheon commented 3 years ago

Try setting storage driver in ~/.config/containers/storage.conf (the commented-out #driver = "overlay" line) to vfs instead of overlay

rugk commented 3 years ago

Well… then I get the "old" error back as it seems… :stuck_out_tongue_winking_eye: (twice, even :smiley: )

Though for some reason the old error is also still displayed for some awkward reason. :laughing:

$ podman version                        
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

storage.conf:

[storage]
driver = "vfs"
#[storage.options]
#mount_program = "/usr/bin/fuse-overlayfs"

mheon commented 3 years ago

Alright. It is complaining that it wants overlay - sorry, must have misread. Looks like we need to change back to overlay and add:

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

That will force the use of fuse-overlayfs, not kernel overlay - rootless users can't mount Overlay (usually - there are a few distros that maintain non-upstreamed patches that allow that) but can mount Fuse FSes, which is the probably cause of your error.

On Sun, Dec 13, 2020 at 5:10 PM rugk notifications@github.com wrote:

Well… then I get the "old" error back as it seems…

$ podman version

ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve

ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve

Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

rugk commented 3 years ago

Ah indeed thanks, now it works.

$ podman version
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

~/.config/containers/storage.conf needs to be like this:

[storage]
driver = "overlay"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"

Still the (duplicated) error messages on this way were confusing. But apart from that it sounds like a config error from downstream.

mheon commented 3 years ago

If we were showing that error with no ~/.config/containers/storage.conf something might be wrong with our baked-in defaults - it should have used those settings automatically...

rhatdan commented 3 years ago

I have a fix for this in containers/storage to explain what is going on.

The problem was that if you did not set a driver, we were not able to get the driver specific fields, so we were not seeing fuse-overlayfs as the mount-program. Now the error would explain that you have to set the driver.

rugk commented 3 years ago

The bug initially happened without any file IIRC. But I see now that if I remove that file, it does work too. If I just comment all lines in the file with # it does show an error though, which is very unexpected.

rhatdan commented 3 years ago

Yes, we expect the user who put the storage.conf into their homedir, to configure the fields that they want. We don't default to override options like mounts_program and mountopts. We do default the driver, and runroot and graphroot.

SahilChaudhary25 commented 3 years ago

Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied I tried this and getting this error now

mheon commented 3 years ago

More details needed - root or rootless, what OS?

SahilChaudhary25 commented 3 years ago

More details needed - root or rootless, what OS?

i want to run it as rootless, but above one i tried with root user.

uname -a Linux test-deployment-8458d6d698-dtqnc 5.4.0-1036-azure #38~18.04.1-Ubuntu SMP Wed Jan 6 18:26:30 UTC 2021 x86_64 Linux

podman version Version: 3.1.0 API Version: 3.1.0 Go Version: go1.16.2 Git Commit: 9a786166922beba71bd8934b709440def82091dc Built: Tue Mar 30 22:46:11 2021 OS/Arch: linux/amd64

Podman info --debug

host: arch: amd64 buildahVersion: 1.20.0 cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: Unknown path: /usr/bin/conmon version: 'conmon version 2.0.27, commit: 3f3bf835bb5bd523d4b4850825dbcbc8a52c0407' cpus: 8 distribution: distribution: alpine version: 3.12.5 eventLogger: file hostname: test-deployment-8458d6d698-dtqnc idMappings: gidmap: null uidmap: null kernel: 5.4.0-1036-azure linkmode: dynamic memFree: 6096883712 memTotal: 33677254656 ociRuntime: name: crun package: Unknown path: /usr/bin/crun version: |- crun version 0.19 commit: e67a75672412975916dac6b87f8346f870e4b99a spec: 1.0.0 +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL os: linux remoteSocket: path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true selinuxEnabled: false slirp4netns: executable: "" package: "" version: "" swapFree: 0 swapTotal: 0 uptime: 35h 59m 3.63s (Approximately 1.46 days) registries: search:

docker.io store: configFile: /etc/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: Unknown Version: |- fuse-overlayfs: version 1.5 fusermount3 version: 3.9.1 FUSE library version 3.9.1 using FUSE kernel interface version 7.31 overlay.mountopt: nodev overlay.skip_mount_home: "true" graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: overlayfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 1 runRoot: /run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 3.1.0 Built: 1617144371 BuiltTime: Tue Mar 30 22:46:11 2021 GitCommit: 9a786166922beba71bd8934b709440def82091dc GoVersion: go1.16.2 OsArch: linux/amd64 Version: 3.1.0

mheon commented 3 years ago

This seems like a new, unrelated bug - please open a new issue. Please include any storage.conf files (/etc/containers/storage.conf, $HOME/.config/containers/storage.conf) when you do.

rhatdan commented 3 years ago

I think this is the issue with rootless overlayfs support.

winston0410 commented 2 years ago

Having the same issue today with Oracle free tier instance. A reboot worked.

fishy commented 2 years ago

I'm seeing the same issue since podman on debian testing upgraded to 3.4.6+ds1-1:

$ podman version
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve 
Version:      3.4.6
API Version:  3.4.6
Go Version:   go1.18.1
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

$ dpkg -s podman
Package: podman
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 34862
Maintainer: Debian Go Packaging Team <pkg-go-maintainers@lists.alioth.debian.org>
Architecture: amd64
Source: libpod
Version: 3.4.6+ds1-1
Depends: libc6 (>= 2.33), libdevmapper1.02.1 (>= 2:1.02.97), libgpgme11 (>= 1.4.1), libseccomp2 (>= 2.4.1), conmon (>= 2.0.18~), containernetworking-plugins (>= 0.8.7), golang-github-containers-common, crun | runc (>= 1.0.0~rc92~)
Recommends: buildah (>= 1.21.0), fuse-overlayfs (>= 1.0.0~), slirp4netns (>= 0.4.1~), catatonit | tini | dumb-init, uidmap, golang-github-containernetworking-plugin-dnsname
Suggests: containers-storage, docker-compose, iptables
Breaks: buildah (<< 1.10.1-6), fuse-overlayfs (<< 0.7.1), slirp4netns (<< 0.4.1)
Conffiles:
 /etc/cni/net.d/87-podman-bridge.conflist a87c090f17c5274af878e7106e969b60
 /etc/containers/libpod.conf ceec5a77b5f6a56d212eeed7b707d322
Description: engine to run OCI-based containers in Pods
 Podman is an engine for running OCI-based containers in Pods.
 Podman provides a CLI interface for managing Pods, Containers, and
 Container Images.
 .
 At a high level, the scope of libpod and podman is the following:
  * Support multiple image formats including the OCI and Docker image
    formats.
  * Support for multiple means to download images including trust & image
    verification.
  * Container image management (managing image layers, overlay filesystems,
    etc).
  * Full management of container lifecycle.
  * Support for pods to manage groups of containers together.
  * Resource isolation of containers and pods.
  * Support for a Docker-compatible CLI interface through Podman.
 .
 Podman is a daemon-less alternative to Docker.
Built-Using: containerd (= 1.6.2~ds1-1), docker-registry (= 2.8.1+ds1-1), docker.io (= 20.10.14+dfsg1-1), golang-1.18 (= 1.18.1-1), golang-dbus (= 5.0.6-1), golang-fsnotify (= 1.5.1-1), golang-ginkgo (= 1.14.2-1), golang-github-acarl005-stripansi (= 0.0~git20180116.5a71ef0-2), golang-github-appc-cni (= 1.0.1-3), golang-github-beorn7-perks (= 1.0.1-1), golang-github-blang-semver (= 4.0.0-1), golang-github-buger-goterm (= 0.0+git20181115.c206103-2), golang-github-cespare-xxhash (= 2.1.1-2), golang-github-checkpoint-restore-go-criu (= 5.3.0-2), golang-github-chzyer-readline (= 1.4.39.g2972be2-3), golang-github-containerd-stargz-snapshotter (= 0.8.0-9), golang-github-containernetworking-plugins (= 1.1.0+ds1-1), golang-github-containers-buildah (= 1.23.1+ds1-2), golang-github-containers-common (= 0.44.5+ds1-1), golang-github-containers-image (= 5.16.0-3), golang-github-containers-ocicrypt (= 1.0.3-1), golang-github-containers-psgo (= 1.5.2-1), golang-github-containers-storage (= 1.37.1+ds1-1), golang-github-coreos-bbolt (= 1.3.5-1), golang-github-coreos-go-iptables (= 0.6.0-1), golang-github-coreos-go-systemd (= 22.3.2-1), golang-github-cyphar-filepath-securejoin (= 0.2.3-1), golang-github-davecgh-go-spew (= 1.1.1-2), golang-github-disiqueira-gotree (= 3.0.2-2), golang-github-docker-docker-credential-helpers (= 0.6.4+ds1-1), golang-github-docker-go-connections (= 0.4.0-3), golang-github-docker-go-metrics (= 0.0.1-1), golang-github-docker-go-units (= 0.4.0-4), golang-github-docker-libtrust (= 0.0~git20150526.0.9cbd2a1-3.1), golang-github-dtylman-scp (= 0.0~git20181017.f3000a3-3), golang-github-fsouza-go-dockerclient (= 1.6.6-1), golang-github-fullsailor-pkcs7 (= 0.0~git20210826.33d0574-2), golang-github-ghodss-yaml (= 1.0.0-1.1), golang-github-go-logr-logr (= 0.4.0-1), golang-github-google-go-cmp (= 0.5.7-1), golang-github-google-go-intervals (= 0.0.2-2), golang-github-google-gofuzz (= 1.2.0-1), golang-github-google-shlex (= 0.0~git20191202.e7afc7f-1), golang-github-google-uuid (= 1.3.0-1), golang-github-gorilla-handlers (= 1.4.2-1), golang-github-gorilla-mux (= 1.8.0-1), golang-github-hashicorp-errwrap (= 1.1.0-1), golang-github-hashicorp-go-multierror (= 1.1.1-1), golang-github-influxdata-tail (= 1.0.0+git20180327.c434825-4), golang-github-ishidawataru-sctp (= 0.0+git20210707.9a39160-1), golang-github-jinzhu-copier (= 0.3.2-2), golang-github-json-iterator-go (= 1.1.12-1), golang-github-juju-ansiterm (= 0.0~git20210929.8b71cc9-1), golang-github-klauspost-compress (= 1.13.6+ds1-2), golang-github-klauspost-pgzip (= 1.2.5-1), golang-github-lunixbochs-vtclean (= 1.0.0-1), golang-github-manifoldco-promptui (= 0.8.0-2), golang-github-mattn-go-colorable (= 0.1.12-1), golang-github-mattn-go-isatty (= 0.0.14-1), golang-github-mattn-go-runewidth (= 0.0.13-2), golang-github-mitchellh-mapstructure (= 1.4.3-1), golang-github-moby-sys (= 0.0~git20220215.b8d8fab-2), golang-github-moby-term (= 0.0~git20210619.3f7ff69-1), golang-github-modern-go-concurrent (= 1.0.3-1.1), golang-github-modern-go-reflect2 (= 1.0.2-2), golang-github-morikuni-aec (= 1.0.0-2), golang-github-opencontainers-go-digest (= 1.0.0-1), golang-github-opencontainers-image-spec (= 1.0.2-2), golang-github-opencontainers-runtime-tools (= 0.9.0+dfsg-2), golang-github-opencontainers-selinux (= 1.10.0+ds1-1), golang-github-opencontainers-specs (= 1.0.2.66.g20a2d97-1), golang-github-openshift-imagebuilder (= 1.2.1+ds1-3), golang-github-pkg-errors (= 0.9.1-2), golang-github-pmezard-go-difflib (= 1.0.0-3), golang-github-proglottis-gpgme (= 0.1.1-1), golang-github-prometheus-client-golang (= 1.11.1-1), golang-github-prometheus-client-model (= 0.2.0-3), golang-github-prometheus-common (= 0.32.1-8), golang-github-prometheus-procfs (= 0.7.3-2), golang-github-rivo-uniseg (= 0.2.0-1), golang-github-safchain-ethtool (= 0.2.0-1), golang-github-spf13-cobra (= 1.4.0-2), golang-github-spf13-pflag (= 1.0.5-2), golang-github-ulikunitz-xz (= 0.5.6-2), golang-github-vbatts-tar-split (= 0.11.1-2), golang-github-vbauerster-mpb (= 7.3.2-1), golang-github-vishvananda-netlink (= 1.1.0.125.gf243826-4), golang-github-vishvananda-netns (= 0.0~git20211101.5004558-1), golang-github-vividcortex-ewma (= 1.1.1-2), golang-github-xeipuuv-gojsonpointer (= 0.0~git20190905.02993c4-2), golang-github-xeipuuv-gojsonreference (= 0.0~git20180127.bd5ef7b-2), golang-github-xeipuuv-gojsonschema (= 1.2.0-3), golang-go-patricia (= 2.3.1-1), golang-go-zfs (= 2.1.1.49.gf784269-1), golang-go.crypto (= 1:0.0~git20220315.3147a52-1), golang-gocapability-dev (= 0.0+git20200815.42c35b4-1), golang-gogoprotobuf (= 1.3.2-1), golang-golang-x-net (= 1:0.0+git20220225.27dd868+dfsg-1), golang-golang-x-sync (= 0.0~git20210220.036812b-1), golang-golang-x-sys (= 0.0~git20211216.1d35b9e-1), golang-golang-x-term (= 0.0~git20210615.6886f2d-1), golang-golang-x-text (= 0.3.7-1), golang-golang-x-xerrors (= 0.0~git20200804.5ec99f8-1), golang-gomega (= 1.10.3-1), golang-google-genproto (= 0.0~git20200413.b5235f6-1), golang-google-grpc (= 1.27.1-1), golang-google-protobuf (= 1.27.1-1), golang-gopkg-inf.v0 (= 0.9.1-1), golang-gopkg-square-go-jose.v2 (= 2.6.0-1), golang-gopkg-tomb.v1 (= 0.0~git20141024.0.dd63297-7), golang-gopkg-yaml.v3 (= 3.0.0~git20210107.496545a-1), golang-goprotobuf (= 1.3.4-2), golang-k8s-klog (= 2.5.0-2), golang-k8s-sigs-structured-merge-diff (= 4.1.2+ds1-1), golang-logrus (= 1.7.0-2), golang-protobuf-extensions (= 1.0.1-1), golang-toml (= 1.0.0-2), golang-yaml.v2 (= 2.4.0-2), rootlesskit (= 1.0.0-1), runc (= 1.1.1+ds1-1)
Homepage: https://github.com/containers/podman

I have no /etc/containers/storage.conf file:

$ sudo ls /etc/containers/
libpod.conf  policy.json  registries.conf  registries.conf.d

and no $HOME/.config/containers/storage.conf either:

$ ls $HOME/.config/containers
ls: cannot access '/home/fishy/.config/containers': No such file or directory

creating ~/.config/containers/storage.conf fixed it for me:

$ cat ~/.config/containers/storage.conf 
[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

$ podman version
Version:      3.4.6
API Version:  3.4.6
Go Version:   go1.18.1
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

so sounds like it's an issue with the default config?

rhatdan commented 2 years ago

Please remove libpod.conf.

fishy commented 2 years ago

After I removed both /etc/containers/libpod.conf and ~/.config/containers/storage.conf I'm back at getting the same error.

Also /etc/containers/libpod.conf is packaged by debian:

$ dpkg -L podman | grep libpod.conf
/etc/containers/libpod.conf

Also I checked the content of /etc/containers/libpod.conf packaged by debian, and I didn't find anything obviously related to the overlay-fs. @rhatdan can you point out which part of libpod.conf could cause this problem?

rhatdan commented 2 years ago

How old is the Podman and Debian?

fishy commented 2 years ago

It upgraded to 3.4.6 earlier today, which looks pretty new to me?

This issue also only started happening after today's upgrade (I can't remember what's the version prior to the upgrade), so this looks like a regression in 3.4.6 to me (or a regression on debian's packaging of podman 3.4.6)

containers / podman

Silverblue: Podman complains kernel not supporting overlay fs (backing file system is unsupported for this graph driver) #8705