search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Podman is not able to create containers with additional image store. #22753

Open mahendra77024 opened 6 months ago

mahendra77024 commented 6 months ago

Issue Description

Hi ,

I have been trying to set up shared location for storing container images. So that, all users on system can leverage these images to create containers rather than pulling container images again. I have followed below blog and able to list the images but while creating containers it's failed with below error.

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

https://www.redhat.com/sysadmin/image-stores-podman

Steps to reproduce the issue

podman --root /var/lib/containers/test-storage pull fedora

chmod -R a+rx /var/lib/containers/test-storage

switch to rootless user

create a storage.conf in ~/.config/containers/storage.conf

[storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you received

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you expected

when run podman run registry.fedoraproject.org/fedora ls -latr / , it should be able to create with container without any errors as image is available in shared location with readonly mode.

podman info output

$podman info
host:
  arch: amd64
  buildahVersion: 1.31.5
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38'
  cpuUtilization:
    idlePercent: 91.79
    systemPercent: 2.31
    userPercent: 5.9
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.9"
  eventLogger: file
  freeLocks: 2045
  hostname: xyz.example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 669
      size: 1
    uidmap:
    - container_id: 0
      host_id: 85617
      size: 1
  kernel: 4.18.0-513.24.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1786728448
  memTotal: 8059088896
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.20.12
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /tmp/podman-run-85617/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 1945104384
  swapTotal: 2147479552
  uptime: 312h 5m 19.00s (Approximately 13.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/users/xx/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/containers/test-storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.12
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
  graphRoot: /home/users/xx/.local/share/containers/storage
  graphRootAllocated: 17169383424
  graphRootUsed: 221696000
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-85617/containers
  transientStore: false
  volumePath: /home/users/xx/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1713524958
  BuiltTime: Fri Apr 19 04:09:18 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa) $ podman version Client: Podman Engine Version: 4.6.1 API Version: 4.6.1 Go Version: go1.20.12 Built: Fri Apr 19 04:09:18 2024 OS/Arch: linux/amd64 $ cat ~/.config/containers/storage.conf [storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$podman info host: arch: amd64 buildahVersion: 1.31.5 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38' cpuUtilization: idlePercent: 91.79 systemPercent: 2.31 userPercent: 5.9 cpus: 2 databaseBackend: boltdb distribution: distribution: '"rhel"' version: "8.9" eventLogger: file freeLocks: 2045 hostname: xyz.example.com idMappings: gidmap:

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

giuseppe commented 6 months ago

how have you pulled the images in the shared store? Have you used fuse-overlays and set force_mask = "shared"?

mahendra77024 commented 6 months ago

Hi @giuseppe ,

I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work

giuseppe commented 5 months ago

have you configured fuse-overlayfs in the storage.conf file?

mahendra77024 commented 5 months ago

Yes I did configure. Please refer postman info and steps to reproduce section for more information. Thanks

mahendra77024 commented 5 months ago

@giuseppe any help on this would be appreciated

snowhanse commented 5 months ago

Seems related to discussion here: https://github.com/containers/podman/issues/19827 ?

mahendra77024 commented 5 months ago

@giuseppe podman doesn't support shared image store for running containers with rootless user?

giuseppe commented 5 months ago

it does, as long as extended attributes are supported by the underlying file system

giuseppe commented 5 months ago

I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work

what storage.conf? /etc/containers/storage.conf?

mahendra77024 commented 5 months ago

@giuseppe I'm running podman as rootless user. So, The storage.conf that it is referring is $HOME/.config/containers/storage.conf

Here is the steps to reproduce issue

podman --root /var/lib/containers/test-storage pull fedora

chmod -R a+rx /var/lib/containers/test-storage

switch to rootless user

create a storage.conf in $HOME/.config/containers/storage.conf

[storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

giuseppe commented 5 months ago

how does the /etc/containers/storage.conf file look like when you pull the image as root?

You need to set force_mask = "shared" there

mahendra77024 commented 5 months ago

@giuseppe , below is the storage.conf file for root. I do see i have already configured force_mask = "shared".

grep -v ^# /etc/containers/storage.conf

[storage]

driver = "overlay"

runroot = "/run/containers/storage"

graphroot = "/var/lib/containers/storage"

[storage.options] ignore_chown_errors = "true" additionalimagestores = ["/var/lib/containers/test-storage"]

pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}

remap-uids = "0:1668442479:65536" remap-gids = "0:1668442479:65536"

[storage.options.overlay] ignore_chown_errors = ""

mount_program = "/usr/bin/fuse-overlayfs"

mountopt = "nodev,metacopy=on"

force_mask = "shared"

[storage.options.thinpool]

giuseppe commented 5 months ago

sorry, I've missed this part earlier:

idMappings:
gidmap:
- container_id: 0
host_id: 669
size: 1
uidmap:
- container_id: 0
host_id: 85617
size: 1

You've only one ID available in your user namespace.

You can try to workaround this limitation using the squash_to_root option for fuse-overlayfs. Try adding it to the mountopt field in the ~/.config/containers/storage.conf file.

mahendra77024 commented 5 months ago

@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?

$ cat ~/.config/containers/storage.conf [storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" mountopt = "squash_to_root" $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest bd78a74aa43c 6 hours ago 233 MB true

$ podman run bd78a74aa43c Error: creating container storage: creating read-write layer with ID "c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff: invalid argument

giuseppe commented 5 months ago

@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?

thanks. The configuration looks correctly, but I don't remember ever testing the combination squash_to_root and force_mode, so it might just be broken

mahendra77024 commented 5 months ago

Thanks @giuseppe. Any insights how this can be addressed.

mahendra77024 commented 4 months ago

@giuseppe is there anything that can be done to resolve this issue ? Thanks

giuseppe commented 4 months ago

can you configure multiple additional IDs for your user?

Is your user listed under /etc/subuid and /etc/subgid?

mahendra77024 commented 4 months ago

Hi @giuseppe , if i map additional ID's for my user that works but we can't use this as we use LDAP and have many other users logging in to the system. that's one of the reason why i'm trying to use shared storage since they fill up disk space.

giuseppe commented 4 months ago

another alternative you've is to rewrite the mode extended attribute and force every file to be owned by root, so its value should be 0:0:$MODE

giuseppe commented 4 months ago

or you can try to use force_mode also for the rootless user, so that the file ownership is forced to root

mahendra77024 commented 4 months ago

@giuseppe Could you please let me know how to do this ? Is this something i need to make a change in storage.conf? Thanks

giuseppe commented 4 months ago

yes, you need to change it in the ~/.config/containers/storage.conf file

mahendra77024 commented 4 months ago

Hi @giuseppe , Could you please share full configuration or example of "force_mode" that has to be set in ~/.config/containers/storage.conf? I couldn't find any document for force_mode in storage.conf. Thanks

giuseppe commented 4 months ago

you've already used force_mode in /etc/containers/storage.conf right? Otherwise the store could not be used by rootless. Now I am suggesting you use the same setting for ~/.config/containers/storage.conf.

[storage]
  driver = "overlay"

[storage.options.overlay]
  mount_program = "/usr/bin/fuse-overlayfs"
  force_mask = "shared"
mahendra77024 commented 4 months ago

@giuseppe , I guess i have already tried this option earlier and also now but i don't see that's working.

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest 85370d566a4c 2 days ago 233 MB true

bd78a74aa43c 8 days ago 233 MB true docker.io/library/alpine latest 1d34ffeaf190 4 weeks ago 8.08 MB true $ cat ~/.config/containers/storage.conf [storage] driver = "overlay" [storage.options] ignore_chown_errors = "true" additionalimagestores = [ "/var/lib/containers/test-storage",] [storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" force_mask = "shared" $ podman run registry.fedoraproject.org/fedora:latest Error: creating container storage: creating read-write layer with ID "8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e/diff: invalid argument
mahendra77024 commented 4 months ago

@giuseppe , If you think this is a bug, can it be added to future request ? Or may be you can try to reproduce this issue from your end that may give you more insights

mahendra77024 commented 3 months ago

Hey @giuseppe , Just following up if you have any suggestions

mahendraredwarasila-snc commented 3 months ago

Hey @giuseppe , Do you have anything to say this can be fixed ?

giuseppe commented 3 months ago

sorry but I am looking at other issues at the moment and I won't have time to look at this one any time soon. If you want to see it fixed, please open a PR. It might need changes in fuse-overlayfs too