search

nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.77k stars 151 forks source link

"OCI runtime attempted to invoke a command that was not found" error from Podman in sysbox system container #682

Open DekusDenial opened 1 year ago

DekusDenial commented 1 year ago

I was trying to follow this post (https://www.redhat.com/sysadmin/podman-inside-kubernetes), and hope to see if I can get Podman working without privileged mode in a K8s pod running Sysbox system container. I understand there are already existing issues regarding Podman integration into Sysbox so I am here to provide more info:

At first I was getting error about /dev/fuse not found and I am aware of the limitation in Sysbox related to this. I knew I need to have it mounted from host , so I was able to workaround by using the io.kubernetes.cri-o.Devices annotation instead of using the kubelet device plugin as mentioned from the post, and made /dev/fuse with permission 0666. However, I then got the following error when trying to start a container via Podman:

sh-5.2# podman  run --rm hello-world
WARN[0000] Found incomplete layer "942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c", deleting it
Error: crun: make `/var/lib/containers/storage/overlay/edd2911b92f517336aa38932e54809852d2f9c6ab718c7df7a3bfb9bdf587b39/merged` private: No such file or directory: OCI runtime attempted to invoke a command that was not found

But, if I create the pod/container with default runc runtime in CRI-O instead, I was able to get pass that error. Of course, the --privileged flag will not work in this case.

apiVersion: v1
kind: Pod
metadata:
  name: podman
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    io.kubernetes.cri-o.Devices: /dev/fuse
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: podman
    image: quay.io/podman/stable
    command: ["sh", "-c", "sleep inf"]
    securityContext:
      capabilities:
        add:
          - "SYS_ADMIN"
          - "MKNOD"
          - "SYS_CHROOT"
          - "SETFCAP"
  restartPolicy: Never

DEBU[0000] Received: -1 DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Error unmounting /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged with fusermount3 - exit status 1 DEBU[0000] Failed to remove mountpoint 942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c overlay: /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged - device or resource busy DEBU[0000] Unmounted container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807" DEBU[0000] Removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 DEBU[0000] Cleaning up container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping... DEBU[0000] Removing all exec sessions for container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 DEBU[0000] Container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 storage is already unmounted, skipping... DEBU[0000] Failed to delete container "a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807": 1 error occurred:

DEBU[0000] unable to remove container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 after failing to start and attach to it: removing container a4cffb913661e7ff137ad055feda57e11546295985182e7d3f44e3335b2c0807 root filesystem: 1 error occurred:

DEBU[0000] ExitCode msg: "crun: make /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged private: no such file or directory: oci runtime attempted to invoke a command that was not found" Error: crun: make /var/lib/containers/storage/overlay/942afa80c440df8217d86da76f739724ec331bac84d0df62efe8413525775e6c/merged private: No such file or directory: OCI runtime attempted to invoke a command that was not found DEBU[0000] Shutting down engines

DekusDenial commented 1 year ago

@rodnymolina @ctalledo any thought?

rodnymolina commented 1 year ago

Hi @DekusDenial, thanks for trying and documenting this effort.

There are a couple of issues to address here before we can support what you are attempting to do:

1) First of all, we need to support rootful podman within a sysbox container, which technically speaking isn't a hard thing to do taking into account where we left off last time we worked on this area -- rootless podman within sysbox would be a totally different story, but I see little value in doing that once the wrapping sysbox container is rootless itself.

2) Extend Sysbox and sysbox-k8s-deploy daemonset support to AL2. As before, this is not rocket science either, but would require cycles that we currently don't have.

In short, it would be difficult for me to give you an ETA for this, but if you are interested, we could help you make these enhancements by your own. Let me know if that's the case.

DekusDenial commented 1 year ago

@rodnymolina thank you for the update. Don’t worry about 2) at all. We are more interested in 1) for rootful in particular and would like help in that even if it means we have to make some changes on our side as we really want sysbox to be the backing runtime for our workload ATM.

DekusDenial commented 1 year ago

@rodnymolina @ctalledo does this sound like something easy to do or you think this needs a second thought? If you point me to where I can make such enhancements, it’d be nice.