search

ES-Nix / podman-rootless

Example of using nix + flakes to have podman rootless working
MIT License
14 stars 0 forks source link

Notes #2

Open PedroRegisPOAR opened 3 years ago

PedroRegisPOAR commented 3 years ago

Some notes

Jessie Frazelle:

image

List:

Cuts:

Texts:

ca-certificates curl less libaudit1 libbrotli1 libbsd0 libbz2-1.0 libcap-ng0 libcom-err2 libcrypt1 libcurl4 libexpat1 libffi7 libfftw3-double3 libfontconfig1 libfreetype6 libgcc-s1 libgcrypt20 libglib2.0-0 libgmp10 libgnutls30 libgomp1 libgpg-error0 libgssapi-krb5-2 libhogweed6 libicu67 libidn2-0 libjpeg62-turbo libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 liblcms2-2 libldap-2.4-2 liblqr-1-0 libltdl7 liblzma5 libmagickcore-6.q16-6 libmagickwand-6.q16-6 libmd0 libmemcached11 libncurses6 libnettle8 libnghttp2-14 libonig5 libp11-kit0 libpam0g libpcre2-8-0 libpcre3 libpng16-16 libpq5 libpsl5 libreadline8 librtmp1 libsasl2-2 libsodium23 libsqlite3-0 libssh2-1 libssl1.1 libstdc++6 libsybdb5 libtasn1-6 libtidy5deb1 libtinfo6 libunistring2 libuuid1 libwebp6 libx11-6 libxau6 libxcb1 libxdmcp6 libxext6 libxml2 libxslt1.1 libzip4 openssl procps zlib1g https://github.com/bitnami/containers/blob/f261c42986fc2f6ed9540aed4d8b03107da9c646/bitnami/wordpress/6/debian-11/Dockerfile#L26

image https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#image-relationships

No matter what we tried, it was nearly impossible to share layers between multiple builds of Polygott. Polygott itself is 20-30GB which adds up quickly when multiple versions are in production at the same time. Will Nix Overtake Docker?

Texts:

The distroless thing

List:

An interesting thing with shellshock so shell-shocked instantaneously made every single web server in the universe that did bash ran faster CGI scripts was vulnerable it could be totally taken over every single web server in the world the only thing to stop that exploit was selinux ok so the day it happened any machine that was running with SELinux and enforcing mode blocked. Refs.:

TODO:

TODO: fix speling erros of transcript: It is actually built as it's complicated we're built it's built as like one big docker container on an Ubuntu base image and then exported into squash FS and distributed via some like binary update scheme and it's just all weird it's it's not any normal technology it was just it was I actually built it at the same time his doctor was building docker for Mac and then we met up at darker calling we were like we built the same thing except mine has no features. Shipit! Presents: How Shopify Uses Nix

TODO: fix speling erros of transcript: shyam us also use nix and production. Not yet I'm hoping that we will by the end of this year once now that we have development environments constructed from from NYX expressions like taking the next step to to turn that into a container is really appealing because typically like especially for rails apps with the whole bundle install step and what not container builds are like the models just don't work together very well so the idea of simply exporting a subgraph of the NYX store as a container is really intensely appealing because it could speed that up a lot and provide a little bit better you know auditability and vulnerability scanning things like this so not yet soon hopefully Shipit! Presents: How Shopify Uses Nix

Note: In rootless Podman, we actually use a fuse-overlayfs executable to create the layer. Rootfull uses the kernel’s overlayfs driver. Currently, the kernel does not allow rootless users to mount overlay filesystems, but they can mount FUSE filesystems. What happens behind the scenes of a rootless Podman container? by Matthew Heon (Red Hat), Dan Walsh (Red Hat), Giuseppe Scrivano (Red Hat)

TODO: Podman in Podman, similar to Docker in Docker?. Related: Daniel Walsh in Is there any chance to run rootless podman container inside another one? "Rootless podman inside of rootless podman, would be very difficult to make work,because of the multiple user namespaces." Related: Red Hat YouTube official channel video, Podman in Podman (Running a Container Within a Container)

TODO: add transcript and explain that for me it is valid to development environments/reproducible environments. Object Oriented Programming vs Functional Programming

An interesting thing with shell shock so shell-shocked instantaneously made every single web server in the universe that did bash ran faster CGI scripts was vulnerable it could be totally taken over every single web server in the world the only thing to stop that exploit was selinux ok so the day it happened any machine that was running with selinux and enforcing mode blocked. A fresh look at SELinux..., start=68&end=95

How to wrap setcap/setuid?

image From: https://github.com/NixOS/nixpkgs/blob/nixos-20.09/pkgs/os-specific/linux/libcap-ng/default.nix#L34 and https://people.redhat.com/sgrubb/libcap-ng/

podman pull docker.io/library/ubuntu:10.04

:bulb: reproduce CVEs using this.

Minimum required kernel version

May 21, 2011: 1 of 5 Dan Walsh, Red Hat, on SELinux security at the RHTUGNL, about history

Nix Cookbook

TODO: podman has this roles somewhere, find it and maybe make a PR correcting it https://github.com/NixOS/nixpkgs/blob/2553aee74fed8c2205a4aeb3ffd206ca14ede60f/nixos/modules/services/cluster/kubernetes/default.nix#L281-L285

podman run docker.io/library/alpine:3.18.3 sh -c "cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 9999"
PedroRegisPOAR commented 3 years ago

Troubleshoot

podman unshare cat /proc/self/uid_map

Refs.:

podman unshare cat /proc/self/uid_map | wc --lines | rg 2

Refs.:

podman unshare env
podman unshare cat /proc/self/mountinfo

TODO: make tests with the strace --failed-only flag to filter!

podman \
unshare \
strace \
--no-abbrev \
--string-limit=4096 \
--follow-forks \
--output=log-strace.txt \
podman \
run \
--log-level=debug \
--rm=true \
--interactive=true \
--tty=true \
docker.io/busybox \
echo 'Hello World!'

From: https://github.com/containers/podman/issues/7004#issuecomment-663888071

nix shell nixpkgs#util-linux
findmnt --submounts /sys/fs/cgroup
nix shell nixpkgs#libselinux
getsebool container_manage_cgroup
selinuxenabled; echo $?
setsebool -P container_manage_cgroup 1

From: 4.2. Starting services within a container using systemd

TODO: https://github.com/containers/podman/blob/main/contrib/podmanimage/stable/Dockerfile

capsh --print

From: https://github.com/containers/podman/issues/6667#issuecomment-648687714

sh-5.0# echo podman:10000:65536 > /etc/subuid
sh-5.0# echo podman:10000:65536 > /etc/subgid

From: https://github.com/containers/podman/issues/6667#issuecomment-648687714

About the ERRO[0057] unable to write pod event: "write unixgram @00017->/run/systemd/journal/socket: sendmsg: no such file or directory". Test in ~/.config/containers/containers.confor /etc/containers/containers.conf

[engine]
events_logger = "file"

From: https://github.com/containers/podman/issues/4325#issuecomment-755314779

We need nix + flake:

For other's, Podman installs from the podman project instructions has repeatedly caused us headaches: Use podman from upstream Ubuntu repo's or don't bother and stick with docker. It has now, for us, reached the point where podman's value add is not enough to justify the additional time. https://github.com/containers/podman/issues/5102#issuecomment-737600718

The error

pr_err("unrecognized mount option \"%s\" or missing value\n", https://github.com/torvalds/linux/blob/8e2e0104ef78d49a51db9acfd24eaf6d52dc779e/fs/overlayfs/super.c#L621

pkgs.fuse-overlayfs # CoW for images, much faster than default vfs https://gist.github.com/adisbladis/187204cb772800489ee3dac4acdd9947#file-podman-shell-nix-L48

Now it gets mount_program /usr/bin/fuse-overlayfs by default, so there is no need for that storage.conf https://github.com/containers/storage/issues/863#issuecomment-805268859

The main message is that native rootless mounts doesn't work with those kernels, and it seems like new installs now know this. https://github.com/containers/storage/issues/863#issuecomment-805631580

if unshare.IsRootless() {
        flags = fmt.Sprintf("%s,userxattr", flags)
}

https://github.com/containers/storage/issues/863#issuecomment-805255089

We don't support the Ubuntu and Debian custom patch for rootless overlay. The main reason is that on old kernels it is not allowed to create whiteout files without an overlay mount and that would complicate how image pulls are performed. https://github.com/containers/storage/pull/816#discussion_r580877045

opts = fmt.Sprintf("%s,userxattr", opts)

https://github.com/containers/storage/commit/6cb8896a97f8f82d6ed86be1051577aceb1fd580#diff-4b0197157e354c66e154f5cdc6fea71c4e43732b4bad32d8652f23bd588e51a1R1029

It is trying to use overlay without fuse-overlays. It works on Ubuntu kernels but something seems to be failing. @rlifshay could you try using fuse-overlays as we do by default on Fedora? https://github.com/containers/buildah/issues/1709#issuecomment-508989292

If I use fuse-overlayfs it works without issue. I am hoping that this can be fixed so native overlay works. That way it theoretically has better performance and I also don't have to manually download and install fuse-overlayfs on Ubuntu. https://github.com/containers/buildah/issues/1709#issuecomment-509046184

Overlayfs is not supported in rootless mode UNLESS fuse-overlay is configured as the mount-program. https://github.com/containers/buildah/issues/1304#issuecomment-458507340

Adapted and works:

[storage]
  driver = "overlay"
  [storage.options]
    mount_program = "/bin/fuse-overlayfs"

https://github.com/containers/buildah/issues/1304#issuecomment-458630736 and https://github.com/containers/podman/issues/2659#issuecomment-544660592

/home/tobwen/podman/usr/local/bin/podman \
--log-level=debug \
--tmpdir /tmp/user/1000/libpod/tmp \
--conmon /home/tobwen/podman/usr/local/bin/conmon \
--network-cmd-path /home/tobwen/podman/usr/local/bin/slirp4netns \
--runtime /home/tobwen/podman/usr/local/bin/runc \
--storage-driver overlay \
--storage-opt "overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs" \
run \
hello-world

https://github.com/containers/podman/issues/2659#issuecomment-544053100

PedroRegisPOAR commented 3 years ago 
stat /dev/kvm
file /dev/kvm
groups
sudo su -c 'groups'
podman run alpine ls /dev
podman run --privileged alpine ls /dev
id -u
stat -c "%U" $(tty)
sudo usermod -a -G kvm $(stat -c "%U" $(tty))

https://askubuntu.com/a/1187716, https://askubuntu.com/a/136804, https://askubuntu.com/a/739132, https://unix.stackexchange.com/a/145751

nix \
shell \
nixpkgs/84aa23742f6c72501f9cc209f29c438766f5352d#podman \
--command \
podman \
run \
--privileged=true \
--device=/dev/fuse \
--device=/dev/kvm \
--env=DISPLAY=':0.0' \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=true \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
docker.nix-community.org/nixpkgs/nix-flakes

nix shell nixpkgs#{qemu,binutils,file,shadow,which,ripgrep}

nix build github:ES-Nix/nix-qemu-kvm/dev#qemu.prepare
sudo chown $(id -u):$(id -g) /dev/kvm

Reboot And the command groups shows the kvm, it could be id.

https://stackoverflow.com/a/64216394

cat /proc/modules | rg -c kvm
sudo chown -R $(id -u):$(id -g) /dev/pts /dev/ptmx

https://github.com/rkt/rkt/issues/2152#issuecomment-216694196 https://github.com/rkt/rkt/issues/3252#issuecomment-253935660

nix \
shell \
nixpkgs/84aa23742f6c72501f9cc209f29c438766f5352d#podman \
--command \
podman \
run \
--privileged=true \
--device=/dev/fuse \
--device=/dev/kvm \
--env=DISPLAY=':0.0' \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=true \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/dev:/dev \
docker.nix-community.org/nixpkgs/nix-flakes
nix \
shell \
nixpkgs/84aa23742f6c72501f9cc209f29c438766f5352d#podman \
--command \
podman \
run \
--privileged=true \
--env=DISPLAY=':0.0' \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=true \
--rm=true \
--security-opt=seccomp=unconfined \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/dev/kvm:/dev/kvm:rw \
docker.nix-community.org/nixpkgs/nix-flakes
stat /dev/kvm
nix shell nixpkgs#findutils --command find /dev -gid 65534
nix shell nixpkgs#{strace,which,ripgrep}
strace -o strace.out -f nix build github:ES-Nix/nix-qemu-kvm/dev#qemu.prepare
cat strace.out | rg -C 10 'Permission denied'

https://stackoverflow.com/questions/32416409/getting-more-info-on-permission-denied


mkdir -p ~/.config/containers
cat << 'EOF' >> ~/.config/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
EOF

mkdir -p ~/.config/containers
cat << 'EOF' >> ~/.config/containers/registries.conf
[registries.search]
registries = ['docker.io']
[registries.block]
registries = []
EOF
nix build nixpkgs/84aa23742f6c72501f9cc209f29c438766f5352d#podman

sudo su 

mkdir -p /etc/containers
cat << 'EOF' >> /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}
EOF
sudo \
result/bin/podman \
run \
--privileged=true \
--env=DISPLAY=':0.0' \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=true \
--rm=true \
--security-opt=seccomp=unconfined \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/dev/kvm:/dev/kvm:rw \
docker.nix-community.org/nixpkgs/nix-flakes

Refs

PedroRegisPOAR commented 3 years ago

Using podman from nix with flakes and test non-ASCII characters

podman \
run \
--interactive=true \
--tty=false \
--rm=true \
--user=0 \
docker.io/nixpkgs/nix-flakes \
<< COMMANDS
echo naïve café
COMMANDS

Executing in interactive mode

podman \
run \
--interactive=true \
--tty=true \
--rm=true \
--user=0 \
docker.io/nixpkgs/nix-flakes

echo naïve café

PedroRegisPOAR commented 3 years ago 
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 1 finished!' \
&& result/bin/podman \
run \
--log-level=debug \
--privileged=true \
--device=/dev/fuse \
--device=/dev/kvm \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=false \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=0755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 2 finished!' \
&& result/bin/podman \
--log-level=debug \
images
COMMANDS
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 1 finished!' \
&& result/bin/podman \
run \
--log-level=debug \
--privileged=true \
--device=/dev/fuse \
--device=/dev/kvm \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=false \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=755 ~/.config/nix && touch ~/.config/nix/nix.conf
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 2 finished!' \
&& result/bin/podman \
--log-level=debug \
pull \
docker.io/library/alpine:3.13.5
COMMANDS
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 1 finished!' \
&& result/bin/podman \
run \
--log-level=debug \
--privileged=true \
--device=/dev/fuse \
--device=/dev/kvm \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--tty=false \
--rm=true \
--user=0 \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
--volume=/etc/localtime:/etc/localtime:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=755 ~/.config/nix && touch ~/.config/nix/nix.conf
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
build \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& echo 'Build 2 finished!' \
&& result/bin/podman \
--log-level=debug \
run \
--log-level=debug \
--interactive=true \
--tty=true \
--rm=true \
docker.io/library/alpine:3.13.5 \
sh \
-c \
'id'
COMMANDS
PedroRegisPOAR commented 2 years ago 
podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=true \
--rm=true \
--userns=host \
--user=0 \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
docker.nix-community.org/nixpkgs/nix-flakes
mkdir --parent --mode=0755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf

mkdir --parent --mode=1777 /var/tmp
nix store gc
du -s /nix/store/
nix profile install nixpkgs#hello
hello
du -s /nix/store/
nix store gc
du -s /nix/store/

nix profile install nixpkgs#hello
hello
du -s /nix/store/
nix store gc
du -s /nix/store/
podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=false \
--rm=true \
--userns=host \
--user=0 \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=0755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
mkdir --parent --mode=1777 /var/tmp
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& podman \
run \
--storage-driver="vfs" \
--cgroups=disabled \
--log-level=error \
--interactive=true \
--network=host \
--tty=true \
alpine \
sh \
-c 'apk add --no-cache curl && echo PinP'
COMMANDS
PedroRegisPOAR commented 2 years ago

3 Levels of nesting podman PinPinP

podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--events-backend=file \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=false \
--rm=true \
--user=0 \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& mkdir -p -m 0755 /var/tmp \
&& podman \
run \
--cgroups=disabled \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--events-backend=file \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--privileged=true \
--tty=false \
--rm=true \
--user=0 \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:rw \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDSNESTED
mkdir --parent --mode=0755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& mkdir --parent --mode=0755 /var/tmp \
&& podman \
run \
--events-backend=file \
--storage-driver="vfs" \
--cgroups=disabled \
--log-level=error \
--interactive=true \
--network=host \
--tty=true \
docker.io/library/alpine:3.14.0 \
sh \
-c 'apk add --no-cache curl && echo PinPinP'
COMMANDSNESTED
COMMANDS
podman \
run \
--env=PATH=/root/.nix-profile/bin:/usr/bin:/bin \
--events-backend=file \
--device=/dev/kvm \
--device=/dev/fuse \
--env="DISPLAY=${DISPLAY:-:0.0}" \
--interactive=true \
--log-level=error \
--network=host \
--mount=type=tmpfs,destination=/var/lib/containers \
--privileged=true \
--tty=false \
--rm=true \
--user=0 \
--volume=/etc/localtime:/etc/localtime:ro \
--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume=/tmp/.X11-unix:/tmp/.X11-unix:ro \
docker.nix-community.org/nixpkgs/nix-flakes \
<<COMMANDS
mkdir --parent --mode=755 ~/.config/nix
echo 'experimental-features = nix-command flakes ca-references ca-derivations' >> ~/.config/nix/nix.conf
nix \
profile \
install \
github:ES-Nix/podman-rootless/from-nixpkgs \
&& mkdir -p -m 0755 /var/tmp \
&& podman  \
--privileged=true \
run \
quay.io/podman/stable \
podman \
run \
--storage-driver="vfs" \
--cgroups=disabled \
ubi8 \
echo \
hello
COMMANDS