Add documentation & better support for HPC/non-root shared `additionalimagestore` updater

[yeswalrus]

Feature request description

In HPC cluster environments, where there are a large number of separate users on separate colocated machines connected to a large shared file store, it is extremely valuable to be able to have an additionalimagecache directory located in the cluster's shared storage. This prevents every machine in the cluster needing to make a local copy, which can be time consuming and wastes space.

HPC clusters are often extremely security sensitive, so it's ideal for the user account responsible for updating the shared cache to not require root access. This is possible to do right now, but is relatively undocumented. Please add official documentation & guidance for the workflow I've discovered, perhaps updating the primary blog post on using additionalimagestores, and support this workflow officially.

Suggest potential solution

I can imagine several improvements to podman pull that would improve this workflow:

• Set permissions as part of pull, removing the need for running chmod -R after on the whole directory
• Do not modify the cache directory if the image already exists
• Allow pulling multiple images in a single command
• When pulling, either split the database or write to a backup file & swap at the end to avoid long periods where the cache is invalidated

Have you considered any alternatives?

Since the core request is to update the documentation & officially support this sort of workflow, the alternative is that you don't do that & we keep using this workflow, and possibly publish my own blogpost.

Additional context

The following python script, run by a non-root account, appears to work in an HPC cluster with a relatively old version of podman (2.2.1):

import sys, subprocess

PODMAN_CACHE_DIR = "/sharefs/podman_cache_0"
PODMAN_CACHE_DIR_BACKUP = "/sharefs/podman_cache_1"
PODMAN_CACHE_DIR_SYMLINK = "/sharefs/podman_cache"

image_names = sys.argv[1:]

uncached_images = [image_name for image_name in image_names if subprocess.run(["podman", "--root", PODMAN_CACHE_DIR, "image", "exists", image_name]).returncode != 0]

if len(uncached_images) == 0:
    print(f"All images ({image_names}) already exist in cache")
    exit(0)

print(f"Adding images to cache: {uncached_images}")

# Begin by updating the symlink to point to the backup dir
# From the time we start pulling the image until the time we finish the chmod, trying to access
# the primary cache will result in failure
if subprocess.run(["ln", "-sTf", PODMAN_CACHE_DIR_BACKUP, PODMAN_CACHE_DIR_SYMLINK]).returncode != 0:
    print("Failed updating symlink")
    exit(-1)

for image_name in uncached_images:
    if subprocess.run(["podman", "--root", PODMAN_CACHE_DIR, "image", "pull", image_name]).returncode != 0:
        print("error while pulling image")
        exit(-1)

print("Images updated, running chmod...")
# Using podman unshare prevents errors when running chmod on the cache directory
if subprocess.run(["podman", "unshare", "chmod", "-Rf", "ag+rx", PODMAN_CACHE_DIR]).returncode != 0:
    print("error while updating permissions on cache dir")
    exit(-1)

# Now that cache update is complete, move the symlink back to the primary cache dir & copy our contents to the backup folder
print("Updating symlink...")
if subprocess.run(["ln", "-sTf", PODMAN_CACHE_DIR, PODMAN_CACHE_DIR_SYMLINK]).returncode != 0:
    print("Failed updating cache symlink")
    exit(-1)

print("Restoring backups...")
if subprocess.run(f"cp -ru {PODMAN_CACHE_DIR}/* {PODMAN_CACHE_DIR_BACKUP}", shell=True).returncode != 0:
    print("Failed updating backup cache")
    exit(-1)

[vrothberg]

Thanks for reaching out, @yeswalrus!

@giuseppe @rhatdan WDYT? I am not sure the chmod -Rf trick will work in all cases, would it?

[rhatdan]

It will require you to use fuse-overlayfs as well, as I understand it.

[giuseppe]

yes and set force_mask="shared" in the storage.conf file

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@yeswalrus Interested in writing the documentation?

[rhatdan]

@giuseppe @vrothberg Should we blog on this? How do setup an additional store to be shared amongst non root users?

[vrothberg]

@giuseppe @vrothberg Should we blog on this? How do setup an additional store to be shared amongst non root users?

Absolutely. Just need to find a new blog site :´(

[rhatdan]

Just use podman.io

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[jharmison-redhat]

Bumping to remove stale - I'd appreciate this blog right about now, even for non-remote filesystems, thanks to how osbuild-composer embeds images for the root user only into /usr/share/containers/storage/overlay-images ...

[martinfg2]

Note that configuring a system-wide login script to set CONTAINERS_STORAGE_CONF=/etc/containers/storage.conf allows that file to act as both rootful and rootless podman storage.conf. However, I'm not part of the Podman/containers development team, so there may be a good reason to have a separate file.

However, the basic problem of needing to chmod -R a+rx every time an image is added remains. If this step is missed, then every other non-root podman user on the system is unable to do any podman function unless they modify their personal storage.conf to remove the additionalimagestores line. Instead, they get an error "...open /overlay-images/images.json: permission denied". For a large multi-user Linux server, this is a big problem.

I second the request to fully document the process, and respectfully request that the chmod be integrated into "podman --root= pull..." so it cannot be missed.

[rhatdan]

If users want to improve the documentation on this, go for it.

@martinfg2 please open a different discussion/issue on getting podman pull support to automatically do the chmod -R a+rx.

[martinfg2]

Done: https://github.com/containers/podman/issues/20826