fix: Also use `storage.conf` when running `rootless` with user `podman`

[gabyx]

The storage.conf is not added to the /home/podman/.config/containers/storage.conf which is a bit odd.

Not sure if that is by design, but generally why should the podman user not have the same setting? The graphroot should probably change so I outcommented it together with runroot which makes them the
default directory podman chooses when runnint rootless.

[cevich]

The storage.conf is not added to the /home/podman/.config/containers/storage.conf which is a bit odd. Not sure if that is by design,

I think this was intentional, the container is meant to be used in either a rootless or rootfull way (see links at bottom). When run rootless, the user will pick up the storage settings from /etc since they're not overridden. In that case, the rootless_storage_path option (default) will be used.

In both cases, I'm not certain about runroot. However it's been set to /run/containers/storage for a really long time in this image so I'm a bit loathe to change it w/o a really really really good reason. I'll see if I can get some clarification on this from the team.

[cevich]

Correction (durrrr): The container can be used both root-ful or root-less but the container user, is intended to be root. I don't think this image was ever meant to also run rootless in the nested podman.

[cevich]

I was told by the team that the paths in the /usr/share/containers/storage.conf are ignored by podman unless overridden in /etc/ or user's home-dir. So I won't be able to accept this PR as-is, it will likely break many people.

One option for you, assuming you have a specific use-case where these changes are needed: Make a derivative container image from this one. Then you can change whatever you like :smiley: Could that be a solution?

[gabyx]

Thanks @cevich for clarifications and sure my own derivative of podman works surely, I thought it might be good to report this issue if it is one, but I need to be more precise:

Running

• podman run -it -u root --privileged --rm quay.io/podman/stable:latest podman info against
• podman run -it -u podman --privileged --rm quoy.io/podman/stable:latest podman info

gives the following diff which I am confused why the difference in the graphOptions are intended?. Thats due to the fact that the configs are picked up differently (?).

4c4,8
<   cgroupControllers: []
---
>   cgroupControllers:
>   - cpu
>   - io
>   - memory
>   - pids
12c16
<     idlePercent: 98.89
---
>     idlePercent: 98.88
23c27
<   hostname: 52f29175332b
---
>   hostname: efeaa7ce4107
25,26c29,48
<     gidmap: null
<     uidmap: null
---
>     gidmap:
>     - container_id: 0
>       host_id: 1000
>       size: 1
>     - container_id: 1
>       host_id: 1
>       size: 999
>     - container_id: 1000
>       host_id: 1001
>       size: 64535
>     uidmap:
>     - container_id: 0
>       host_id: 1000
>       size: 1
>     - container_id: 1
>       host_id: 1
>       size: 999
>     - container_id: 1000
>       host_id: 1001
>       size: 64535
30c52
<   memFree: 58279739392
---
>   memFree: 58303004672
49c71
<       rundir: /run/crun
---
>       rundir: /tmp/podman-run-1000/crun
65c87
<     path: /run/podman/podman.sock
---
>     path: /tmp/podman-run-1000/podman/podman.sock
69c91
<     rootless: false
---
>     rootless: true
85c107
<   uptime: 0h 13m 49.00s
---
>   uptime: 0h 13m 41.00s
107c129
<   configFile: /etc/containers/storage.conf
---
>   configFile: /home/podman/.config/containers/storage.conf
114,125c136,137
<   graphOptions:
<     overlay.imagestore: /usr/lib/containers/storage
<     overlay.mount_program:
<       Executable: /usr/bin/fuse-overlayfs
<       Package: fuse-overlayfs-1.13-1.fc39.x86_64
<       Version: |-
<         fusermount3 version: 3.16.1
<         fuse-overlayfs: version 1.13-dev
<         FUSE library version 3.16.1
<         using FUSE kernel interface version 7.38
<     overlay.mountopt: nodev,fsync=0
<   graphRoot: /var/lib/containers/storage
---
>   graphOptions: {}
>   graphRoot: /home/podman/.local/share/containers/storage
130c142
<     Native Overlay Diff: "false"
---
>     Native Overlay Diff: "true"
132c144
<     Supports shifting: "true"
---
>     Supports shifting: "false"
138c150
<   runRoot: /run/containers/storage
---
>   runRoot: /tmp/containers-user-1000/containers
140c152
<   volumePath: /var/lib/containers/storage/volumes
---
>   volumePath: /home/podman/.local/share/containers/storage/volumes

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.77
    systemPercent: 0.5
    userPercent: 0.72
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: 8a9d868a5e05
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.19
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 58273886208
  memTotal: 67342225408
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 77309407232
  swapTotal: 77309407232
  uptime: 0h 11m 51.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc39.x86_64
      Version: |-
        fusermount3 version: 3.16.1
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.1
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1049659179008
  graphRootUsed: 181540704256
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 1708357294
  BuiltTime: Mon Feb 19 15:41:34 2024
  GitCommit: ""
  GoVersion: go1.21.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

where as the podman run -it -u podman --privileged --rm quay.io/podman/stable:latest podman info

shows differences in graphOptions:

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.84
    systemPercent: 0.48
    userPercent: 0.68
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: 88ac140ab26f
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.6.19
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 58301374464
  memTotal: 67342225408
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 77309407232
  swapTotal: 77309407232
  uptime: 0h 12m 56.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 1049659179008
  graphRootUsed: 181541183488
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 1708357294
  BuiltTime: Mon Feb 19 15:41:34 2024
  GitCommit: ""
  GoVersion: go1.21.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

[cevich]

gives the following diff

Looking through the diff, they all seem reasonable given the execution context.

[gabyx]

@cevich :

gives the following diff

Looking through the diff, they all seem reasonable given the execution context.

Why is the overlay.mount_program: not set when running as podman but when set as root? Can fuse-overlayfs only be used when run as root?

[rhatdan]

mount_program should only be set if the user wants to use a different Overlay then the kernel overlay, this should seldom be set, unless user intends to use fuseoverlayfs.

[rhatdan]

@giuseppe PTAL

[gabyx]

mount_program should only be set if the user wants to use a different Overlay then the kernel overlay, this should seldom be set, unless user intends to use fuseoverlayfs.

Ok, I understand: normally when mount_program is not set, podman tries to use the kernel overlayfs thing which is the direct implementation of the overlayfs functionality in the kernel...

[giuseppe]

I don't think we need this change.

We need to keep the storage for podman and root separate, images stored for root have a different mapping than for the podman user

[gabyx]

@giuseppe : Would you be so kind to explain where my understanding problem lies: The graphroot and runroot should not be copied I agree, thats also in the PR changes, but the graphOptions should be the same correct and thats what my change does. Its probably not so nice by copying the storage.conf from root and adjusting it by commenting the values! . I could only take the graphOptions for sure I guess...

if

graphOptions:
     overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
       Executable: /usr/bin/fuse-overlayfs
       Package: fuse-overlayfs-1.13-1.fc39.x86_64
       Version: |-
         fusermount3 version: 3.16.1
         fuse-overlayfs: version 1.13-dev
         FUSE library version 3.16.1
         using FUSE kernel interface version 7.38
     overlay.mountopt: nodev,fsync=0

is in the storage.conf for root it should also be for podman , but it is currently not , IMO.

[giuseppe]

you can't use additionalimage with rootless unless you are sure it was pulled with force_mask

[gabyx]

force_mask

:confused: Sorry, I do not understand. If you actually wanna help: Please reread my question and my last post (what has this to do with additionalimage which is not even a thing, I guess you mean additionalimagestores.... Beeing precise even for green horns on these topics would help and not result in a back and forth in this discussion.

[cevich]

I can confirm this change breaks rootless use of the podman container.