docker compatiple with runc runtime

[mozartilize]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue: Case 1:

1. Having docker, podman installed

2. export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

3. Run docker run --rm -p 8080:80 nginx

Describe the results you received: Getting error with message unable to upgrade to tcp, received 409

Case 2:

1. Remove docker, only have podman

2. export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

3. Run nomad nomad agent -dev, start a job which runs nginx with docker driver

job "example" {
  datacenters = ["dc1"]
  type        = "service"

  group "nginx" {
    network {
      port "http" {
        to = 80
      }
    }

    task "nginx" {
      driver = "docker"

      config {
        image = "nginx"
        ports = ["http"]
      }

      resources {
        cpu    = 300
        memory = 256
      }
    }
  }
}

Describe the results you received:

[DEBUG] client.driver_mgr.docker: failed to start container: driver=docker container_id=be4c6fe2f05e94a9b9ee19d801b7b6b3b6ef5f67da0d80723fa3491493266976 attempt=1 error="API error (500): runc: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: OCI runtime error"

Additional information you deem important (e.g. issue happens only occasionally):

Before I run podman system reset, both 2 cases return the same error as case 2.

Output of podman version:

podman 4.2.1-1 on Manjaro Linux

Output of podman info:

Client:       Podman Engine
Version:      4.2.1
API Version:  4.2.1
Go Version:   go1.19
Git Commit:   62b324ddf718411b1d4d0ba8117c632f7f984a38-dirty
Built:        Thu Sep  8 13:52:54 2022
OS/Arch:      linux/amd64

Output of docker info:

Click me

``` Client: Context: default Debug Mode: false Plugins: compose: Docker Compose (Docker Inc., 2.11.2) Server: Containers: 2 Running: 0 Paused: 0 Stopped: 0 Images: 1 Server Version: 4.2.1 Storage Driver: overlay Native Overlay Diff: true Using metacopy: false Backing Filesystem: extfs Supports d_type: true Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge macvlan Log: k8s-file none passthrough journald Swarm: inactive Runtimes: runsc crun kata krun runc runj Default Runtime: runc Init Binary: containerd version: runc version: init version: Security Options: seccomp Profile: default Kernel Version: 5.15.74-3-MANJARO Operating System: manjaro OSType: linux Architecture: amd64 CPUs: 12 Total Memory: 15.53GiB Name: witzer ID: 4cab533e-3fdc-4522-801a-b87395e605fb Docker Root Dir: /home/mozart/.local/share/containers/storage Debug Mode: false Experimental: true Live Restore Enabled: false Product License: Apache-2.0 WARNING: No swap limit support WARNING: No cpu cfs quota support WARNING: No cpu cfs period support WARNING: No cpu shares support WARNING: No cpuset support WARNING: IPv4 forwarding is disabled WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled ``` **Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)** Yes/No **Additional environment details (AWS, VirtualBox, physical, etc.):**

[mheon]

Please provide a full podman info

[mozartilize]

@mheon I provided it in the description, is there any information I miss?

[mheon]

That is podman version not podman info

[mozartilize]

@mheon my bad

Click here

``` host: arch: amd64 buildahVersion: 1.27.0 cgroupControllers: - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: /usr/bin/conmon is owned by conmon 1:2.1.4-1 path: /usr/bin/conmon version: 'conmon version 2.1.4, commit: bd1459a3ffbb13eb552cc9af213e1f56f31ba2ee' cpuUtilization: idlePercent: 99.7 systemPercent: 0.2 userPercent: 0.1 cpus: 12 distribution: distribution: manjaro version: unknown eventLogger: journald hostname: ... idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 5.15.74-3-MANJARO linkmode: dynamic logDriver: journald memFree: 15381032960 memTotal: 16676130816 networkBackend: netavark ociRuntime: name: runc package: /usr/bin/runc is owned by runc 1.1.4-1 path: /usr/bin/runc version: |- runc version 1.1.4 spec: 1.0.2-dev go: go1.19 libseccomp: 2.5.4 os: linux remoteSocket: exists: true path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /etc/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.4 swapFree: 0 swapTotal: 0 uptime: 0h 5m 39.00s plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: search: - docker.io store: configFile: ~/.config/containers/storage.conf containerStore: number: 2 paused: 0 running: 0 stopped: 2 graphDriverName: overlay graphOptions: {} graphRoot: ~/.local/share/containers/storage graphRootAllocated: 250584948736 graphRootUsed: 201386381312 graphStatus: Backing Filesystem: extfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 3 runRoot: /run/user/1000/containers volumePath: ~/.local/share/containers/storage/volumes version: APIVersion: 4.2.1 Built: 1662619974 BuiltTime: Thu Sep 8 13:52:54 2022 GitCommit: 62b324ddf718411b1d4d0ba8117c632f7f984a38-dirty GoVersion: go1.19 Os: linux OsArch: linux/amd64 Version: 4.2.1 ```

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mozartilize Are you still seeing this issue with latest podman?

[mozartilize]

@rhatdan yes, with podman 4.3.1, the issue still happens

[rhatdan]

@mheon Thoughts?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Thoughts?

[rhatdan]

@mozartilize did you ever try this with crun?

[mozartilize]

@rhatdan yes, with crun it works

$ cat ~/.config/containers/containers.conf
[engine]
runtime = "crun"

$ docker run --rm -p 8080:80 nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/07 07:55:45 [notice] 1#1: using the "epoll" event method
2023/01/07 07:55:45 [notice] 1#1: nginx/1.23.2
2023/01/07 07:55:45 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2023/01/07 07:55:45 [notice] 1#1: OS: Linux 5.15.85-1-lts
2023/01/07 07:55:45 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
2023/01/07 07:55:45 [notice] 1#1: start worker processes
2023/01/07 07:55:45 [notice] 1#1: start worker process 24
2023/01/07 07:55:45 [notice] 1#1: start worker process 25
2023/01/07 07:55:45 [notice] 1#1: start worker process 26
2023/01/07 07:55:45 [notice] 1#1: start worker process 27
2023/01/07 07:55:45 [notice] 1#1: start worker process 28
2023/01/07 07:55:45 [notice] 1#1: start worker process 29
2023/01/07 07:55:45 [notice] 1#1: start worker process 30
2023/01/07 07:55:45 [notice] 1#1: start worker process 31
2023/01/07 07:55:45 [notice] 1#1: start worker process 32
2023/01/07 07:55:45 [notice] 1#1: start worker process 33
2023/01/07 07:55:45 [notice] 1#1: start worker process 34
2023/01/07 07:55:45 [notice] 1#1: start worker process 35

[rhatdan]

@AkihiroSuda @kolyshkin @giuseppe Any ideas?

[AkihiroSuda]

Is the case 2 relevant to the case 1? If not probably it's better to open another issue.

[mozartilize]

@AkihiroSuda as I mentioned,

Before I run podman system reset, both 2 cases return the same error as case 2.

But I have an update after some testing today that the second case doesn't happen anymore.

The only problem is when I run docker run --rm -p 8080:80 nginx with DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock and runc engine, I get unable to upgrade to tcp, received 409

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[giuseppe]

could you look in the journal if there is any useful information why it failed?

Could you try dropping the resources block in the compose file?

[pini-gh]

Hi,

I'm not sure if my issue is closely related to this one. Please tell me if I need to open a new one.

I have this error unable to upgrade to tcp, received 409 (exec.go:71:0s) when using piplr as a gitlab-runner:

Preparing the "docker" executor 00:02
Using Docker executor with image quay.io/podman/stable ...
Pulling docker image quay.io/podman/stable ...
Using docker image sha256:f58758d0b5823a50b52ebc24ce4c4a615652a73c879bd69d40f27ae832e2fb15 for quay.io/podman/stable with digest quay.io/podman/stable@sha256:05d6df73cb7303e4e5ba0d2a7c6dcba8add16bb7c7acd6d36d267dafed6b8b75 ...
Preparing environment 00:00
ERROR: Job failed (system failure): prepare environment: unable to upgrade to tcp, received 409 (exec.go:71:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Is is weird because when starting the runner manually with podman start pipglr it works fine. The above error occurs when the runner is started with systemctl --user start pipglr.service, which just runs the very same podman start pipglr command against the very same pipglr container.

I've compared the results of running podman --debug info in both contexts and saw no differences beside volatile values such as dates or sizes.

My current pipglr image has podman v4.6.1. An older image I created a few months ago with podman 4.4.1 works flawlessly.

The image is built against centos:stream9. Is there any easy way to grab podman releases between v4.4.1 and v4.6.1 for CentOS Stream 9 so I could bisect against podman version? I cannot find any release older than 4.6.0-0.3.el9 anymore.

[mozartilize]

One thing that I forgot to mention that I managed to run both docker and podman in rootless mode.

[pini-gh]

I have this error unable to upgrade to tcp, received 409 (exec.go:71:0s) when using piplr as a gitlab-runner:

This was https://github.com/containers/podman/issues/19930. Downgrading crun to 1.8.7 works as a workaround until the fix is released.