podman system reset fails when switching from vfs to overlay

[asottile]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Extending from #7396 -- while switching from vfs to overlay podman system reset is unable to reset state due to an error in changing the graph driver

Steps to reproduce the issue:

1. run any container with the vfs graph driver

2. set ~/.config/containers/storage.conf to:

   [storage]
   driver = "overlay"
   [storage.options]
   mount_program = "/usr/bin/fuse-overlayfs"

3. Run podman system reset

Describe the results you received:

$ podman system reset
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve 
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve 
Error: vfs driver does not support overlay.mount_program options

Describe the results you expected:

A successful reset

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.0.4
API Version:  1
Go Version:   go1.14.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 5
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: babibox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-42-generic
  linkmode: dynamic
  memFree: 6446026752
  memTotal: 8348299264
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 1964396544
  swapTotal: 1964396544
  uptime: 10m 5.43s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 2.0.4~1 amd64 [installed]
podman/unknown 2.0.4~1 arm64
podman/unknown 2.0.4~1 armhf
podman/unknown 2.0.4~1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

virtualbox, but I also saw the same behaviour on an aws instance

[rhatdan]

@ashley-cui PTAL

[vrothberg]

@ashley-cui, did you find time to look into the issue?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I am not sure if this is possible, the more I think about it. I think you would need to do this in two steps.

First reset the storage.

podman system reset

Now edit the storage.conf And run a container.

The problem with changing the storage.conf first, is you confuse podman system reset, into attempting to reset and overlay storage and this blows up pointing back to the original vfs storage.

Probably need to fix the documentation on podman system reset to explain how to do this.