Open mahendra77024 opened 6 months ago
how have you pulled the images in the shared store? Have you used fuse-overlays and set force_mask = "shared"
?
Hi @giuseppe ,
I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work
have you configured fuse-overlayfs
in the storage.conf
file?
Yes I did configure. Please refer postman info and steps to reproduce section for more information. Thanks
@giuseppe any help on this would be appreciated
Seems related to discussion here: https://github.com/containers/podman/issues/19827 ?
@giuseppe podman doesn't support shared image store for running containers with rootless user?
it does, as long as extended attributes are supported by the underlying file system
I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work
what storage.conf
? /etc/containers/storage.conf
?
@giuseppe I'm running podman as rootless user. So, The storage.conf that it is referring is $HOME/.config/containers/storage.conf
Here is the steps to reproduce issue
switch to rootless user
create a storage.conf in $HOME/.config/containers/storage.conf
[storage] driver = "overlay"
[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]
[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true
$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument
how does the /etc/containers/storage.conf
file look like when you pull the image as root?
You need to set force_mask = "shared"
there
@giuseppe , below is the storage.conf file for root. I do see i have already configured force_mask = "shared".
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options] ignore_chown_errors = "true" additionalimagestores = ["/var/lib/containers/test-storage"]
pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}
remap-uids = "0:1668442479:65536" remap-gids = "0:1668442479:65536"
[storage.options.overlay] ignore_chown_errors = ""
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,metacopy=on"
force_mask = "shared"
[storage.options.thinpool]
sorry, I've missed this part earlier:
idMappings:
gidmap:
- container_id: 0
host_id: 669
size: 1
uidmap:
- container_id: 0
host_id: 85617
size: 1
You've only one ID available in your user namespace.
You can try to workaround this limitation using the squash_to_root
option for fuse-overlayfs
. Try adding it to the mountopt
field in the ~/.config/containers/storage.conf
file.
@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?
$ cat ~/.config/containers/storage.conf [storage] driver = "overlay"
[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]
[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" mountopt = "squash_to_root" $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest bd78a74aa43c 6 hours ago 233 MB true
$ podman run bd78a74aa43c Error: creating container storage: creating read-write layer with ID "c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff: invalid argument
@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?
thanks. The configuration looks correctly, but I don't remember ever testing the combination squash_to_root
and force_mode
, so it might just be broken
Thanks @giuseppe. Any insights how this can be addressed.
@giuseppe is there anything that can be done to resolve this issue ? Thanks
can you configure multiple additional IDs for your user?
Is your user listed under /etc/subuid
and /etc/subgid
?
Hi @giuseppe , if i map additional ID's for my user that works but we can't use this as we use LDAP and have many other users logging in to the system. that's one of the reason why i'm trying to use shared storage since they fill up disk space.
another alternative you've is to rewrite the mode extended attribute and force every file to be owned by root, so its value should be 0:0:$MODE
or you can try to use force_mode
also for the rootless user, so that the file ownership is forced to root
@giuseppe Could you please let me know how to do this ? Is this something i need to make a change in storage.conf? Thanks
yes, you need to change it in the ~/.config/containers/storage.conf
file
Hi @giuseppe , Could you please share full configuration or example of "force_mode" that has to be set in ~/.config/containers/storage.conf? I couldn't find any document for force_mode in storage.conf. Thanks
you've already used force_mode
in /etc/containers/storage.conf
right? Otherwise the store could not be used by rootless. Now I am suggesting you use the same setting for ~/.config/containers/storage.conf
.
[storage]
driver = "overlay"
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
force_mask = "shared"
@giuseppe , I guess i have already tried this option earlier and also now but i don't see that's working.
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest 85370d566a4c 2 days ago 233 MB true
@giuseppe , If you think this is a bug, can it be added to future request ? Or may be you can try to reproduce this issue from your end that may give you more insights
Hey @giuseppe , Just following up if you have any suggestions
Hey @giuseppe , Do you have anything to say this can be fixed ?
sorry but I am looking at other issues at the moment and I won't have time to look at this one any time soon. If you want to see it fixed, please open a PR. It might need changes in fuse-overlayfs too
Issue Description
Hi ,
I have been trying to set up shared location for storing container images. So that, all users on system can leverage these images to create containers rather than pulling container images again. I have followed below blog and able to list the images but while creating containers it's failed with below error.
$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument
https://www.redhat.com/sysadmin/image-stores-podman
Steps to reproduce the issue
podman --root /var/lib/containers/test-storage pull fedora
chmod -R a+rx /var/lib/containers/test-storage
switch to rootless user
create a storage.conf in ~/.config/containers/storage.conf
[storage] driver = "overlay"
[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]
[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true
$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument
Describe the results you received
$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument
Describe the results you expected
when run podman run registry.fedoraproject.org/fedora ls -latr / , it should be able to create with container without any errors as image is available in shared location with readonly mode.
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true
$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa) $ podman version Client: Podman Engine Version: 4.6.1 API Version: 4.6.1 Go Version: go1.20.12 Built: Fri Apr 19 04:09:18 2024 OS/Arch: linux/amd64 $ cat ~/.config/containers/storage.conf [storage] driver = "overlay"
[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]
[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"
$podman info host: arch: amd64 buildahVersion: 1.31.5 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38' cpuUtilization: idlePercent: 91.79 systemPercent: 2.31 userPercent: 5.9 cpus: 2 databaseBackend: boltdb distribution: distribution: '"rhel"' version: "8.9" eventLogger: file freeLocks: 2045 hostname: xyz.example.com idMappings: gidmap:
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting