search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
24.02k stars 2.43k forks source link

`docker-compose`: Passing gpu with `driver: cdi` is not supported #19338

Open raldone01 opened 1 year ago

raldone01 commented 1 year ago

Issue Description

May be related to https://github.com/containers/podman/issues/19330. See also https://github.com/NVIDIA/nvidia-container-toolkit/issues/126.

CC @elezar

CDI Support in Docker will be an experimental feature in the Docker 25 release.

It will support the following docker-compose.yml file:

version: '3.8'
services:
cdi_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
           - driver: cdi
             device_ids:
                - nvidia.com/gpu=all
                - nvidia.com/gds=all
                - nvidia.com/mofed=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L

Steps to reproduce the issue

Steps to reproduce:

Prerequisites:

Observe the following docker-compose.yml:

version: '3.8'
services:
  nvidia_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu, utility, compute]
    command:
      - bash
      - -c
      - |
        nvidia-smi -L
  runtime_test: # this is working
    image: ubuntu:20.04
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L
  cdi_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
           - driver: cdi
             device_ids:
                - nvidia.com/gpu=all
                - nvidia.com/gds=all
                - nvidia.com/mofed=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L

Launch via docker-compose up.

Describe the results you received

Only the runtime container works.

Describe the results you expected

All three containers should show the same output.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.7-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 98.59
    systemPercent: 0.38
    userPercent: 1.02
  cpus: 72
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: argon
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.4.4-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 54521659392
  memTotal: 135022321664
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 42947567616
  swapTotal: 42947567616
  uptime: 21h 35m 45.00s (Approximately 0.88 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/main/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/main/.local/share/containers/storage
  graphRootAllocated: 857601998848
  graphRootUsed: 271414001664
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/main/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685139594
  BuiltTime: Sat May 27 00:19:54 2023
  GitCommit: 9eef30051c83f62816a1772a743e5f1271b196d7-dirty
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

nicoya commented 1 year ago

Is this expected to work when using podman-compose? None of the three containers in the above docker-compose.yml seem to work for me, however I can get nVidia support in a container via a plain podman commandline:

$ podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2070 (UUID: GPU-336c5a8a-839f-69a8-08cf-e3fdcd01e019)
raldone01 commented 1 year ago

Interesting. for me the runtime_test is working.

Thing is I had the understanding that podman-docker should be a fully compatible emulation so I consider this an issue.

Since nvidia.com/gpu cdi driver is supported the docker gpu api should be mapped to the podman cdi api in podman-docker.

nicoya commented 1 year ago

Got it working with this:

version: '3.8'
services:
  nvidia_test:
    image: ubuntu
    devices:
      - nvidia.com/gpu=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L
itzsimpl commented 1 year ago

For me, none of these docker-compose.yml approaches works, however I can get nVidia support in a container via a plain podman commandline.

raldone01 commented 1 year ago

nvidia_test: image: ubuntu devices:

  • nvidia.com/gpu=all command:
  • bash
  • -c
  • | nvidia-smi -L

When I try this I get the following: Error response from daemon: container create: stat nvidia.com/gpu=all: no such file or directory

What version of podman are you running? Do you have any special configuration active?

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

elezar commented 1 year ago

@raldone01 if you could point me at the code for podman-compose, I could try to see what additions are required to add CDI support there. I would assume that this would need similar extensions to those made to the --device command line argument. Your error message indicates that the entries in devices: are being interpreted as device nodes.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25. In addition, the mechanism to request these device in the compose YAML do not match your expectations.

raldone01 commented 1 year ago

if you could point me at the code for podman-compose, I could try to see what additions are required to add CDI support there.

What do you want me to point you at? The code for the podman-compose implementation and its argument parser? I myself am not familiar with the code base. I could clone the repo and do some searches for you but I am not sure what code you are referring to. Please clarify.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25. In addition, the mechanism to request these device in the compose YAML do not match your expectations.

Can you show me the correct yaml format? If possible please link me to documentation outlining the correct planned yaml format. I tried to search for it and found lots of conflicting information.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25.

I know and mentioned it in the issue description.

arielnmz commented 1 year ago

Can confirm too that none of the services presented in the OP work for me @raldone01 did you configure the container runtime using the docker instructions? The NVIDIA CTK guide says that CDI is recommended for podman

raldone01 commented 1 year ago

I configured the runtime using sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid. Afterwards podman containers with sudo podman run -ti --rm --device=nvidia.com/gpu=all ubuntu:18.04 nvidia-smi -L worked. I am pretty sure this uses cdi.

I will look into the link you sent on Friday.

arielnmz commented 1 year ago

@raldone01 okay but that is using podman only, right? normal podman commands seem to work fine with CDI, but how did you manage to make docker-compose work using CDI?

raldone01 commented 1 year ago

Yes this is using podman commands directly. This issue is exactly about cdi not working with podman-compose. Maybe this wasn't clear. I was not able to get cdi to work with podman-compose.

Cdi works fine with podman directly. Nvidia Runtime (legacy) works with podman-compose but will be or is deprecated.

raldone01 commented 1 year ago

If anyone has a working podman/docker-compose yml with cdi please share it. As far as I know it is not supported.

arielnmz commented 1 year ago

@raldone01 sorry I got confused by my own post. I am curious how you got the runtime_test part working. I do have the runtime installed and configured (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker) but I just get nvidia-smi command not found

arielnmz commented 1 year ago

Aight, just to follow up on this issue, I added the nvidia runtime to the podman (containers) configuration and set it as the default:

$cat  /usr/share/containers/containers.conf
runtime = "nvidia"
...
[engine.runtimes]
...
nvidia = [
  "/usr/bin/nvidia-container-runtime",
]

So this works:

$ podman --log-level DEBUG system service -t0
...
Using OCI runtime "/usr/bin/nvidia-container-runtime"
...

This works, too:

$ podman run --rm  --security-opt=label=disable nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi -L
...
GPU 0: NVIDIA RTX A5000 (UUID: GPU-82e4260a-3604-4c94-4abf-3ecccd9b28f8)

But tried running the runtime_test service from the OP and still get:

$ docker-compose up runtime_test
...
/opt/nvidia/nvidia_entrypoint.sh: line 67: exec: nvidia-smi: not found

podman info reports using the nvidia runtime:

...
  ociRuntime:
    name: nvidia
    package: nvidia-container-toolkit-base-1.14.2-1.x86_64
    path: /usr/bin/nvidia-container-runtime
    version: |-
      NVIDIA Container Runtime version 1.14.2
      commit: 807c87e057e13fbd559369b8fd722cc7a6f4e5bb
      spec: 1.1.0-rc.2

      runc version 1.1.8
      commit: v1.1.8-0-g82f18fe
      spec: 1.0.2-dev
      go: go1.19.11
      libseccomp: 2.5.3
...
raldone01 commented 1 year ago

As far as I can tell all options in /usr/share/containers/containers.conf are commented out and I can't remember changing it. I will try to get a reproduce script working using distrobox or something like that. My default ociRuntime in podman info is crun. I just verified docker is not even installed on the machine. I am using Docker Compose version 2.20.3.

Does sudo docker run --rm -e NVIDIA_VISIBLE_DEVICES=all ubuntu nvidia-smi -L work for you?

❯ sudo docker run --rm -e NVIDIA_VISIBLE_DEVICES=all ubuntu nvidia-smi -L
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
GPU 0: NVIDIA T600 (UUID: GPU-STUFF)
GPU 1: NVIDIA GeForce GTX 1080 (UUID: GPU-STUFF)

Without env:

❯ sudo docker run --rm ubuntu nvidia-smi -L
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

The command you provided does not work on my machine as I use crun as the default runtime for podman.

❯ sudo podman run --rm  --security-opt=label=disable ubuntu nvidia-smi -L
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Try to do the following (pulled from my command history):

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid
sudo nvidia-ctk runtime configure --runtime=podman
sudo nano /etc/nvidia-container-runtime/config.toml
no-cgroups = false
runtimes = [
  "docker-runc",
  "runc",
  "crun"
]
arielnmz commented 1 year ago

I think the culprit might be that I'm not using compose v2. I was able to get this working, however, by using podman-compose and passing run args to the call:

podman-compose --podman-run-args '--device nvidia.com/gpu=all --security-opt=label=disable' run --rm runtime_test

And this works too:

podman-compose --podman-run-args '--device nvidia.com/gpu=all --security-opt=label=disable' up runtime_test

This works like a charm:

$ podman-compose run --rm runtime_test

version: "3.8"

services:
  runtime_test: # this is working
    image: nvidia/cuda:12.2.0-runtime-ubuntu22.04
    devices:
     - nvidia.com/gpu=all
    security_opt:
      - "label=disable"
    command: nvidia-smi -L
rhatdan commented 1 year ago

Can we close this issue now, since there is a workaround, or is there still something Podman needs to fix?

arielnmz commented 1 year ago

@rhatdan I think the original issue remains: device reservations is broken using podman-docker and it works using normal docker

rhatdan commented 1 year ago

Does it work with rootful podman? Is this more a case of rootless versus rootful?

raldone01 commented 1 year ago

I got a bit of new information. I'm currently reworking the issue. I will provide a reproducible Distrobox script to better test the issue. I have not tried it with rootless yet. The issue still exists. Should I update the issue itself or add a new comment.

Edit: You can track my progress here: https://github.com/raldone01/podman-docker-gpu-issue.git Distrobox was not the right tool for the job. The reproduce script will be using lxd and qemu.

rhatdan commented 1 year ago

Add a new comment.

mrnerdhair commented 11 months ago

I am experiencing the same issue here, using the standalone docker-compose 2.23.0. I can do sudo podman run -ti --rm --device=nvidia.com/gpu=all ubuntu:18.04 nvidia-smi -L successfully, but none of the above flavors of compose file work.

jesulo commented 10 months ago

Hi, is the some option for enable gpu using podman-compose.yml? regards

ageisen2000 commented 9 months ago

I am still experiencing this issue.

gautaz commented 8 months ago

Hello,

Here is the current behavior I am experiencing:

❯ podman version
Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.21.7
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64

❯ podman compose version
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

Docker Compose version 2.24.6

❯ podman run --rm --device nvidia.com/gpu=all -it docker.io/library/ubuntu nvidia-smi
Fri Mar  8 15:28:56 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A500 Laptop GPU     Off |   00000000:01:00.0 Off |                  N/A |
| N/A   37C    P8              4W /   30W |     232MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

❯ podman compose run testgpu
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

unable to upgrade to tcp, received 409
Error: executing /home/del/.nix-profile/bin/docker-compose run testgpu: exit status 1

❯ podman run --rm -it docker.io/library/ubuntu nvidia-smi
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Here is the compose file used:

services:
  testgpu:
    image: docker.io/library/ubuntu
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              device_ids:
                - nvidia.com/gpu=all

Is this related to this issue or is it another issue ?

elezar commented 8 months ago

@gautaz what docker compose does is map the devices section in the compose file to HostConfig modifications (specifically DeviceRequest entries) that are interpreted by the Docker daemon. I am not sure how this interacts with Podman.

If someone could point me at the execution path from running podman-compose -- and how Podman is invoked -- we can identify where support for the cdi driver needs to be added.

laurensV commented 8 months ago

Maybe related to my problem: I am trying to use the Docker API to create a container on a remote podman server with GPU support. With docker this can be done with device requests, but this comment https://github.com/containers/podman/pull/21180#issuecomment-1927171385 suggests that podman is not handling data via device_requests.. How would I use the docker api to enable gpu support in podman?

rhatdan commented 8 months ago

Have you tried with podman 5.0, some fixes whent in to handle GPUs from remote API.

gautaz commented 8 months ago

@elezar @laurensV The way docker-compose/podman-compose are supposed to communicate with podman is in fact also unclear to me. As @elezar pointed out, docker-compose communicates with docker through the daemon API but, in my particular case, I am using podman in its original rootless design, which means, AFAIU, no daemon, no API access through a socket.

@rhatdan I guess your question was for @laurensV. In the meantime, I have also tried podman 5 but I am hitting other difficulties that I need to fully understand. These difficulties are related to the fact that since version 5, podman uses pasta for its network layer and not slirp4netns anymore. The fact that the Linux distribution I am using (NixOS) does not support pasta adds another layer of difficulties (this seems to be a work in progress though).

qdm12 commented 8 months ago

Any news on this? Podman 5.x.x on Ubuntu is for now unusable / too-unstable as @gautaz mentioned.

So back to using Podman 4.9.3, I did a few experiments using a docker-compose.yml, all with the common body:

services:
  issue-19338:
    image: ubuntu
    entrypoint: /bin/bash -c "ls /dev/ | grep nvidia"

where ls /dev/ | grep nvidia shows nvidia devices if the passthrough is successful. CDI definitions are generated using sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml.

The tries and results are in the following table:

Compose program YML setting NVIDIA devices passthrough?
podman-compose devices block Success
podman-compose deploy block with nvidia driver Fail
podman-compose deploy block with cdi driver Fail
compose devices block Fail: container won't start Error response from daemon: container create: stat nvidia.com/gpu=0: no such file or directory
compose devices block + privileged: true Fail
compose deploy block with nvidia driver Fail
compose deploy block with cdi driver Fail

For more details:

Conclusions:

gautaz commented 7 months ago

Hello,

A little follow-up since podman 5 has reached unstable:

❯ podman version
Client:       Podman Engine
Version:      5.0.2
API Version:  5.0.2
Go Version:   go1.22.2
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64

❯ docker compose version
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

Docker Compose version 2.26.1

❯ ls /dev | grep nvidia
nvidia0
nvidia-caps
nvidiactl
nvidia-modeset
nvidia-uvm
nvidia-uvm-tools

❯ podman run --rm --device nvidia.com/gpu=all ubuntu ls /dev | grep nvidia
nvidia-modeset
nvidia-uvm
nvidia-uvm-tools
nvidia0
nvidiactl

❯ cat compose.yaml
services:
  testgpu:
    image: ubuntu
    command: sh -c "ls /dev | grep nvidia"
    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              device_ids:
                - nvidia.com/gpu=all

❯ docker compose up
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

[+] Running 1/0
 ✔ Container test-testgpu-1  Created                                                                                                 0.0s
Attaching to testgpu-1
testgpu-1 exited with code 1

So still no access to the GPU with compose/podman through CDI.

loopback-kr commented 6 months ago

This test was run inside a Slurm cluster with podman support. There will be differences from an on-premises server, but hopefully this will help.

hyunseoki@Master:~/projects/calc-pi$ srun --pty --gpus 2  bash
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

hyunseoki@gpu2:~/projects/calc-pi$ docker version
Client:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.18.1
Built:        Thu Jan  1 09:00:00 1970
OS/Arch:      linux/amd64
hyunseoki@gpu2:~/projects/calc-pi$ docker-compose version
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.2
podman-compose version 1.0.6
podman --version 
podman version 4.6.2
exit code: 0
hyunseoki@gpu2:~/projects/calc-pi$ nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
hyunseoki@gpu2:~/projects/calc-pi$ docker run -it --rm --device nvidia.com/gpu=all python nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
hyunseoki@gpu2:~/projects/calc-pi$ cat /etc/containers/pre-exec-hooks/000_nvidia_cdi.sh 
#!/bin/bash
if [ -e /usr/bin/nvidia-ctk ]; then
  if [ -e /var/run/cdi/nvidia.yaml ]; then
    sleep $(( (RANDOM % 6) + 2 ))
    nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
  else
    nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
  fi
fi
exit 0
hyunseoki@gpu2:~/projects/calc-pi$ cat docker-compose.yml 
services:
  vscode:
    image: docker.io/python
    # build:
      # dockerfile: vscode.Dockerfile
    tty: true
    stdin_open: true
    shm_size: 1TB
    ports:
      - $FORWARD_PORT:8888
    devices:
      - nvidia.com/gpu=all
    volumes:
      - $PWD:/workspace
    environment:
      - SLURMD_NODENAME=$SLURMD_NODENAME
      - PASSWORD=1
      - SUBDOMAIN=$USER
      - FORWARD_PORT=$FORWARD_PORT
    command: nvidia-smi -L
hyunseoki@gpu2:~/projects/calc-pi$ docker-compose up
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.2
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=calc-pi', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
recreating: ...
** excluding:  set()
podman stop -t 10 calc-pi_vscode_1
calc-pi_vscode_1
exit code: 0
podman rm calc-pi_vscode_1
calc-pi_vscode_1
exit code: 0
recreating: done

['podman', 'network', 'exists', 'calc-pi_default']
podman create --name=calc-pi_vscode_1 --label io.podman.compose.config-hash=a7dda14fa6087ec8f2e0557b2bae6ed958ed5468c68034ee6c61d734c7b3e456 --label io.podman.compose.project=calc-pi --label io.podman.compose.version=1.0.6 --label PODMAN_SYSTEMD_UNIT=podman-compose@calc-pi.service --label com.docker.compose.project=calc-pi --label com.docker.compose.project.working_dir=/home/hyunseoki/projects/calc-pi --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=vscode --device nvidia.com/gpu=all -e SLURMD_NODENAME=gpu2 -e PASSWORD=1 -e SUBDOMAIN=hyunseoki -e FORWARD_PORT=20580 -v /home/hyunseoki/projects/calc-pi:/workspace --net calc-pi_default --network-alias vscode -p 20580:8888 --shm-size 1TB -i --tty docker.io/python nvidia-smi -L
7ab542be0eb79b26afd526a70955c7ae34b08d74c878b816ac73813d6bdf35d0
exit code: 0
podman start -a calc-pi_vscode_1
[vscode] | GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
[vscode] | GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
exit code: 0
sreevatsank1999 commented 4 months ago

What is the TL;DR ?

is it that we cannot use docker-compose as the external compose provider for podman? (given that we need gpu support)

is there a chance that this will be fixed in the near future?

rhatdan commented 4 months ago

Podman does not implement the client side of compose on the server side, so it requires either docker-compose or podman-compose (a separate repository) to process the compose syntax and generate podman containers.

sreevatsank1999 commented 4 months ago

Podman does not implement the client side of compose on the server side, so it requires either docker-compose or podman-compose (a separate repository) to process the compose syntax and generate podman containers.

Snooping on the socket connection /run/user/3938/podman/podman.sock using socat, I find that docker-compose makes the following API calls to the podman system service API server, and I think it might be possible to handle this correctly at the podman API server.

docker-compose + deploy block makes the following API call:

POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2561\r
Content-Type: application/json\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-187626433197f294-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": null,
                "DeviceCgroupRules": null,
                "DeviceRequests": [
                    {
                        "Driver": "",
                        "Count": 1,
                        "DeviceIDs": null,
                        "Capabilities": [
                            [
                                "gpu"
                            ]
                        ],
                        "Options": null
                    }
                ],
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }

docker-compose + device block API call:

POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2577\r
Content-Type: application/json\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-18aadb2ff5c0b9ec-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "4a70b70543a273e8ee7f45bf50783bcd3aece1f5ae5bbfba74f61d41f5bcc59b",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [
                    {
                        "PathOnHost": "nvidia.com/gpu=0",
                        "PathInContainer": "nvidia.com/gpu=0",
                        "CgroupPermissions": "rwm"
                    }
                ],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }

Full request/response trace captures:

`docker-compose` + deploy block capture ```http > 2024/07/31 18: 56: 53.457915 length=101 from=0 to=100 HEAD /_ping HTTP/1.1\r Host: api.moby.localhost\r User-Agent: Docker-Client/unknown-version (linux)\r \r < 2024/07/31 18: 56: 53.458332 length=318 from=0 to=317 HTTP/1.1 200 OK\r Api-Version: 1.41\r Builder-Version: \r Buildkit-Version: \r Cache-Control: no-cache\r Docker-Experimental: true\r Libpod-Api-Version: 5.1.2\r Libpod-Buildah-Version: 1.36.0\r Ostype: linux\r Pragma: no-cache\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0000fc548\r Date: Thu, 01 Aug 2024 01: 56: 53 GMT\r \r > 2024/07/31 18: 56: 53.459053 length=153 from=101 to=253 GET /v1.41/info HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-04b12442401aad2a-01\r \r > 2024/07/31 18: 56: 54.467236 length=297 from=0 to=296 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-2dc839f2dc7ce974-01\r \r < 2024/07/31 18: 56: 54.473446 length=1696 from=0 to=1695 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 54 GMT\r Content-Length: 1480\r \r [ { "Id": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722475833, "Ports": [], "Labels": { "com.docker.compose.config-hash": "0e4992f78ef02e6d2b4d2586b96e15393e23549641a57cbbcd3a5d9b45061d29", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "State": "created", "Status": "Created", "NetworkSettings": { "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "bec3219e5b3c" ], "MacAddress": "", "NetworkID": "test_default", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null } } }, "Mounts": [], "Name": "", "Config": null, "NetworkingConfig": null, "Platform": null, "DefaultReadOnlyNonRecursive": false } ] > 2024/07/31 18: 56: 54.476142 length=167 from=297 to=463 GET /v1.41/images/ubuntu/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-5bf8faacf8205821-01\r \r < 2024/07/31 18: 56: 54.482639 length=1976 from=1696 to=3671 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 54 GMT\r Content-Length: 1760\r \r { "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "RepoTags": [ "docker.io/library/ubuntu:latest" ], "RepoDigests": [ "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30", "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c" ], "Parent": "", "Comment": "", "Created": "2024-06-07T12:00:09.099611108Z", "ContainerConfig": { "Hostname": "35a88802559", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "DockerVersion": "24.0.5", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/bin/bash" ], "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" } }, "Architecture": "amd64", "Os": "linux", "OsVersion": "24.0.5", "Size": 80563381, "VirtualSize": 80563381, "GraphDriver": { "Data": { "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work" }, "Name": "overlay" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" }, "Container": "" } > 2024/07/31 18: 56: 54.486734 length=170 from=464 to=633 GET /v1.41/networks/test_default HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-9efda98ca27b3e24-01\r \r < 2024/07/31 18: 56: 54.491938 length=785 from=3672 to=4456 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 54 GMT\r Content-Length: 570\r \r { "Name": "test_default", "Id": "72257744b1a3a1880aa5397dbc01e790027fe57a942c08c1355d7c2c888725ca", "Created": "2024-07-31T18:30:32.237867878-07:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": { "driver": "host-local" }, "Config": [ { "Subnet": "10.89.1.0/24", "Gateway": "10.89.1.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": { "com.docker.compose.network": "default", "com.docker.compose.project": "test", "com.docker.compose.version": "2.29.1" } } > 2024/07/31 18: 56: 54.494158 length=156 from=634 to=789 GET /v1.41/version HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-388d483afddbd5fe-01\r \r < 2024/07/31 18: 56: 54.543292 length=2712 from=318 to=3029 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0000fc548\r Date: Thu, 01 Aug 2024 01: 56: 54 GMT\r Transfer-Encoding: chunked\r \r 800\r { "ID": "5a8824e0-b67f-4973-9335-777669ab056e", "Containers": 10, "ContainersRunning": 0, "ContainersPaused": 0, "ContainersStopped": 8, "Images": 7, "Driver": "overlay", "DriverStatus": [ [ "Backing Filesystem", "btrfs" ], [ "Supports d_type", "true" ], [ "Native Overlay Diff", "true" ], [ "Using metacopy", "false" ], [ "Supports shifting", "false" ], [ "Supports volatile", "true" ] ], "Plugins": { "Volume": [ "local" ], "Network": [ "bridge", "macvlan", "ipvlan" ], "Authorization": null, "Log": [ "k8s-file", "none", "passthrough", "journald" ] }, "MemoryLimit": true, "SwapLimit": false, "CpuCfsPeriod": false, "CpuCfsQuota": false, "CPUShares": false, "CPUSet": false, "PidsLimit": true, "IPv4Forwarding": true, "BridgeNfIptables": true, "BridgeNfIp6tables": true, "Debug": false, "NFd": 15, "OomKillDisable": false, "NGoroutines": 15, "SystemTime": "2024-07-31T18:56:54.543021208-07:00", "LoggingDriver": "", "CgroupDriver": "systemd", "NEventsListener": 0, "KernelVersion": "6.5.0-44-generic", "OperatingSystem": "ubuntu", "OSVersion": "22.04", "OSType": "linux", "Architecture": "amd64", "IndexServerAddress": "", "RegistryConfig": { "AllowNondistributableArtifactsCIDRs": [], "AllowNondistributableArtifactsHostnames": [], "InsecureRegistryCIDRs": [], "IndexConfigs": {}, "Mirrors": [] }, "NCPU": 64, "MemTotal": 472804831232, "GenericResources": null, "DockerRootDir": "/home/groberts/.local/share/containers/storage", "HttpProxy": "", "HttpsProxy": "", "NoProxy": "", "Name": "wcsng-36", "Labels": null, "ExperimentalBuild": true, "ServerVersion": "5.1.2", "Runtimes": { "crun": { "path": "/usr/bin/crun" }, "crun-vm": { "path": "/usr/bin/crun-vm" }, "crun-wasm": { "path": "/usr/bin/crun-wasm" }, "kata": { "path": "/usr/bin/kata-runtime" }, "krun": { "path": "/usr/bin/krun" }, "ocijail": { "path": "/usr/local/bin/ocijail" }, "runc": { "path": "/usr/bin/runc" }, "runj": { "path": "/usr/local/bin/runj" }, "runsc": { "path": "/usr/bin/runsc" }, "youki": { "path": "/usr/local/bin/youki" } }, "DefaultRuntime": "crun", "Swarm": { "NodeID": "", "NodeAddr": "", "LocalNodeState": "inactive", "ControlAvailable": false, "Error": "", "RemoteManagers": null }, "LiveRestoreEnabled": false, "Isolation": "", "InitBinary": "", "ContainerdCommit": { "ID": "","Expec\r1a7\r ted":""},"RuncCommit":{"ID":"","Expected":""},"InitCommit":{"ID":"","Expected":""},"SecurityOptions":["name=apparmor","name=seccomp,profile=default","name=rootless"],"ProductLicense":"Apache-2.0","CDISpecDirs":null,"Warnings":[],"BuildahVersion":"1.36.0","CPURealtimePeriod":false,"CPURealtimeRuntime":false,"CgroupVersion":"2","Rootless":true,"SwapFree":0,"SwapTotal":0,"Uptime":"1h 39m 40.00s (Approximately 0.04 days)"} \r 0\r \r 2024/07/31 18: 56: 54 socat[ 80523 ] E write(6, 0x638cd71cc000, 2712): Broken pipe < 2024/07/31 18: 56: 55.550681 length=1108 from=4457 to=5564 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 893\r \r { "Platform": { "Name": "linux/amd64/ubuntu-22.04" }, "Components": [ { "Name": "Podman Engine", "Version": "5.1.2", "Details": { "APIVersion": "5.1.2", "Arch": "amd64", "BuildTime": "1969-12-31T16:00:00-08:00", "Experimental": "false", "GitCommit": "", "GoVersion": "go1.22.5", "KernelVersion": "6.5.0-44-generic", "MinAPIVersion": "4.0.0", "Os": "linux" } }, { "Name": "Conmon", "Version": "conmon version 2.1.2, commit: ", "Details": { "Package": "conmon_100:2.1.2~0_amd64" } }, { "Name": "OCI Runtime (crun)", "Version": "crun version 1.15\\ncommit: e6eacaf4034e84185fd8780ac9262bbf57082278\\nrundir: /run/user/3141/crun\\nspec: 1.0.0\\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL", "Details": { "Package": "crun_100:1.15-1_amd64" } } ], "Version": "5.1.2", "ApiVersion": "1.41", "MinAPIVersion": "1.24", "GitCommit": "", "GoVersion": "go1.22.5", "Os": "linux", "Arch": "amd64", "KernelVersion": "6.5.0-44-generic", "BuildTime": "1969-12-31T16:00:00-08:00" } > 2024/07/31 18: 56: 55.552874 length=167 from=790 to=956 GET /v1.41/images/ubuntu/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-b9516f77fec95279-01\r \r < 2024/07/31 18: 56: 55.555169 length=1976 from=5565 to=7540 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 1760\r \r { "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "RepoTags": [ "docker.io/library/ubuntu:latest" ], "RepoDigests": [ "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30", "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c" ], "Parent": "", "Comment": "", "Created": "2024-06-07T12:00:09.099611108Z", "ContainerConfig": { "Hostname": "35a88802559", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "DockerVersion": "24.0.5", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/bin/bash" ], "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" } }, "Architecture": "amd64", "Os": "linux", "OsVersion": "24.0.5", "Size": 80563381, "VirtualSize": 80563381, "GraphDriver": { "Data": { "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work" }, "Name": "overlay" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" }, "Container": "" } > 2024/07/31 18: 56: 55.560339 length=2819 from=957 to=3775 POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Content-Length: 2561\r Content-Type: application/json\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-187626433197f294-01\r \r { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": [ "nvidia-smi" ], "Image": "ubuntu", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1" }, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "", "Config": null }, "NetworkMode": "test_default", "PortBindings": {}, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 0, 0 ], "CapAdd": null, "CapDrop": null, "CgroupnsMode": "", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": [], "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 0, "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": [ { "Driver": "", "Count": 1, "DeviceIDs": null, "Capabilities": [ [ "gpu" ] ], "Options": null } ], "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "NetworkingConfig": { "EndpointsConfig": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338" ], "MacAddress": "", "DriverOpts": null, "NetworkID": "", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DNSNames": null } } } } < 2024/07/31 18: 56: 55.588605 length=307 from=7541 to=7847 HTTP/1.1 201 Created\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 88\r \r { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Warnings": [] } > 2024/07/31 18: 56: 55.589584 length=229 from=3776 to=4004 GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-03167f6a8e160f24-01\r \r < 2024/07/31 18: 56: 55.592408 length=4585 from=7848 to=12432 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Transfer-Encoding: chunked\r \r 84b\r { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Created": "2024-08-01T01:56:55.574628083Z", "Path": "nvidia-smi", "Args": [ "nvidia-smi" ], "State": { "Status": "created", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "0001-01-01T00:00:00Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "", "Name": "/bec3219e5b3c_test-issue-19338-1", "RestartCount": 0, "Driver": "overlay", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": [], "HostConfig": { "Binds": [], "ContainerIDFile": "", "LogConfig": { "Type": "journald", "Config": null }, "NetworkMode": "bridge", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 0, 0 ], "CapAdd": [], "CapDrop": [], "CgroupnsMode": "", "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": [], "GroupAdd": [], "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "private", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [], "UTSMode": "private", "UsernsMode": "", "ShmSize": 65536000, "Runtime": "oci", "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "user.slice", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": 0, "OomKillDisable": false, "PidsLimit": 2048, "Ulimits": [ { "Name": "RLIMIT_NPROC", "Hard": 4194304, "Soft": 4194304 } ], "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "LowerDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff"\r 800\r , "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/work" }, "Name": "overlay" }, "SizeRootFs": 0, "Mounts": [], "Config": { "Hostname": "f4bbe1f14929", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=podman" ], "Cmd": [ "nvidia-smi" ], "Image": "docker.io/library/ubuntu:latest", "Volumes": null, "WorkingDir": "/", "Entrypoint": [], "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "StopSignal": "15", "StopTimeout": 10 }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "SandboxKey": "", "Ports": {}, "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "f4bbe1f14929" ], "MacAddress": "", "NetworkID":"test_defa\r a7\r ult","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DriverOpts":null,"DNSNames":null}}}} \r 0\r \r > 2024/07/31 18: 56: 55.604440 length=249 from=4005 to=4253 POST /v1.41/containers/bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300/stop HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Content-Length: 0\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-ba74fc813d6bbf81-01\r \r < 2024/07/31 18: 56: 55.607485 length=170 from=12433 to=12602 HTTP/1.1 204 No Content\r Api-Version: 1.41\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r \r > 2024/07/31 18: 56: 55.608029 length=227 from=4254 to=4480 DELETE /v1.41/containers/bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300 HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-a2578f73f9e824eb-01\r \r < 2024/07/31 18: 56: 55.614576 length=170 from=12603 to=12772 HTTP/1.1 204 No Content\r Api-Version: 1.41\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r \r > 2024/07/31 18: 56: 55.615211 length=275 from=4481 to=4755 POST /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/rename?name=test-issue-19338-1 HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Content-Length: 0\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-0462c738d7e536cd-01\r \r < 2024/07/31 18: 56: 55.618356 length=170 from=12773 to=12942 HTTP/1.1 204 No Content\r Api-Version: 1.41\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r \r > 2024/07/31 18: 56: 55.619196 length=402 from=4756 to=5157 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%2C%22com.docker.compose.service%3Dissue-19338%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-4eba62b7d00a0032-01\r \r < 2024/07/31 18: 56: 55.621533 length=1792 from=12943 to=14734 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 1576\r \r [ { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722477415, "Ports": [], "Labels": { "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "State": "created", "Status": "Created", "NetworkSettings": { "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "f4bbe1f14929" ], "MacAddress": "", "NetworkID": "test_default", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null } } }, "Mounts": [], "Name": "", "Config": null, "NetworkingConfig": null, "Platform": null, "DefaultReadOnlyNonRecursive": false } ] > 2024/07/31 18: 56: 55.626484 length=229 from=5158 to=5386 GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-db0f741e43607a1c-01\r \r < 2024/07/31 18: 56: 55.627999 length=4572 from=14735 to=19306 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Transfer-Encoding: chunked\r \r 83e\r { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Created": "2024-08-01T01:56:55.574628083Z", "Path": "nvidia-smi", "Args": [ "nvidia-smi" ], "State": { "Status": "created", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 0, "Error": "", "StartedAt": "0001-01-01T00:00:00Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "", "Name": "/test-issue-19338-1", "RestartCount": 0, "Driver": "overlay", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": [], "HostConfig": { "Binds": [], "ContainerIDFile": "", "LogConfig": { "Type": "journald", "Config": null }, "NetworkMode": "bridge", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 0, 0 ], "CapAdd": [], "CapDrop": [], "CgroupnsMode": "", "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": [], "GroupAdd": [], "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "private", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [], "UTSMode": "private", "UsernsMode": "", "ShmSize": 65536000, "Runtime": "oci", "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "user.slice", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": 0, "OomKillDisable": false, "PidsLimit": 2048, "Ulimits": [ { "Name": "RLIMIT_NPROC", "Hard": 4194304, "Soft": 4194304 } ], "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "LowerDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff"\r 800\r , "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/work" }, "Name": "overlay" }, "SizeRootFs": 0, "Mounts": [], "Config": { "Hostname": "f4bbe1f14929", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=podman" ], "Cmd": [ "nvidia-smi" ], "Image": "docker.io/library/ubuntu:latest", "Volumes": null, "WorkingDir": "/", "Entrypoint": [], "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "StopSignal": "15", "StopTimeout": 10 }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "SandboxKey": "", "Ports": {}, "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "f4bbe1f14929" ], "MacAddress": "", "NetworkID":"test_defa\r a7\r ult","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DriverOpts":null,"DNSNames":null}}}} \r 0\r \r > 2024/07/31 18: 56: 55.640836 length=347 from=0 to=346 POST /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/attach?stderr=1&stdin=1&stdout=1&stream=1 HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Content-Length: 0\r Connection: Upgrade\r Content-Type: text/plain\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-853e06bb9b4d2ad1-01\r Upgrade: tcp\r \r < 2024/07/31 18: 56: 55.888321 length=561 from=0 to=560 HTTP/1.1 409 Conflict\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0009db718\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 340\r \r { "cause": "OCI runtime attempted to invoke a command that was not found", "message": "preparing container f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd for attach: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found", "response": 409 } > 2024/07/31 18: 56: 55.889157 length=262 from=5387 to=5648 GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/logs?follow=1&stderr=1&stdout=1&tail= HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-85cf00e37654625c-01\r \r < 2024/07/31 18: 56: 55.891360 length=181 from=19307 to=19487 HTTP/1.1 200 OK\r Api-Version: 1.41\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 0\r \r > 2024/07/31 18: 56: 55.891696 length=300 from=5649 to=5948 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-e007044560aca1d3-01\r \r > 2024/07/31 18: 56: 55.892578 length=402 from=0 to=401 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%2C%22com.docker.compose.service%3Dissue-19338%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-2885149299a36c61-01\r \r < 2024/07/31 18: 56: 55.894293 length=1828 from=19488 to=21315 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 55 GMT\r Content-Length: 1612\r \r [ { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722477415, "Ports": [], "Labels": { "com.docker.compose.config-hash":"13075ac5dc2afadcb2de1ca22e075f593c12b18< 2024/07/31 18:56:55.894780 length=1828 from=0 to=18275HdT6TdP3/b14.515 3270d01 0O1Ka\ra 4Abp1if-7V0e4rds"i,o"nc:o m1..d4o1c\rk eCro.nctoemnpto-sTey.pceo:n taapipnleirc-antuimobne/rj"s:o"n1\r",L"icbopmo.dd-oAcpkie-rV.ecrosmipoons:e .5d.e1p.e2n\rd sS_eornv"e:r": " ,L"icbopmo.dd/o5c.k1e.r2. c(olmipnousxe).\ri mXa-gRee"f:e"rsehnac2e5-6I:d3: 5 a08x8c800020595d9ad5d3280\r7 7Dea5t8e4: 3 9T4h4u7, 1 d0d1a aA1uag2 c250b2f4d 10618: 9536b: 85259 eGaM5T7\r6 1C9o3n0t1eenbt3-9L0e8n"g,t"hc:o m1.6d1o2c\rk e\rr .[c{o"mIpdo"s:e".fo4nbeboef1ff"1: 4"9F2a9lasee6"0, 2"6c4o1m5.4d5o5cck3e2r2.5cao6mdp6o6see4.6parao1j9efc7t4"1: 5"0t3e6sdtd"d, 2"acaoamb.9d1oac2kdecrd."c,o"mNpaomsees."p:r[o"j/etcets.tc-oinsfsiuge_-f1i9l3e3s8"-: 1""/ ]h,o"mIem/aggreo"b:e"rdtosc/kdeerv./idoo/clkiebrr/atreys/tu/bduonctkue:rl-actoemspto"s,e".Iymmalg"e,I"Dc"o:m".sdhoac2k5e6r:.3c5oam8p8o8s0e2.5p5r9odjde2c0t7.7weo5r8k4i3n9g4_4d7i1rd"d:a"a/1hao2mce5/bgfrdo1b6e8r9t3sb/8d2e9ve/ad5o7c6k1e9r3/0t1eesbt3"9, 0"8c"o,m".Cdoomcmkaenrd."c:o"mnpvoisdei.ar-espmlia"c,e""C:r"ebaetce3d2"1: 91e752b234c797b43105f, 4"4P8o1r1t3s3"b:e[a ]1,c"cL6a6bee3lds3"7: 0{ 8"6c7o3mb.9d6occ0kbebr0.3c1o4mdp7o8sbe4.ecdoan3f0i0g"-,h"acsohm".:d"o1c3k0e7r5.acco5mdpco2saef.asdecrbv2idcee1"c:a"2i2ses0u7e5-f1599333c812"b, 1"8c5odm6.dd3obc4k5e5r3.7cdo1m0p1oasae4.bv1efr7s0i4odn"",:""c2o.m2.9d.o1c"k,e"ro.rcgo.moppoesnec.ocnotnatianienresr.-inmuamgbee.rr"e:f".1n"a,m"ec"o:m".udboucnkteur".,c"oomrpgo.soep.ednecpoenntdasi_noenr"s:."i"m,a"gceo.mv.edroscikoenr".:c"o2m4p.o0s4e". }i,m"aSgtea"t:e""s:h"ac2r5e6a:t3e5da"8, 8"8S0t2a5t5u9sd"d: 2"0C7r7eea5t8e4d3"9, 4"4N7e1tdwdoarak1Sae2tct5ibnfgds1"6: 8{ 9"3Nbe8t2w9oerak5s7"6: 1{ 9"3t0e1setb_3d9e0f8a"u,l"tc"o:m{."dIoPcAkMeCro.ncfoimgp"o:sneu.loln,e"oLfifn"k:s""F:anluslel",,""Acloima.sdeosc"k:e[r".tceosmtp-oisses.uper-o1j9e3c3t8"-: 1""t,e"sits"s,u"ec-o1m9.3d3o8c"k,e"rf.4cbobmep1ofs1e4.9p2r9o"j ]e,c"tM.accoAndfdirge_sfsi"l:e"s1"a: : "f/ch:o8m0e:/2g3r:o1b5e:rdt9s"/,d"eNve/tdwoocrkkeIrD/"t:e"stte/sdto_cdkeefra-uclotm"p,o"sEen.dypmoli"n,t"IcDo"m:."d"o,c"keGra.tceowmapyo"s:e".1p0r.o8j9e.c1t..1w"o,r"kIiPnAgd_ddriers"s: "":/"h1o0m.e8/9g.r1o.b6e"r,t"sI/PdPerve/fdioxcLkeenr"/:t2e4s,t""I,P"vc6oGma.tdeowcakye"r:."c"o,m"pGolsoeb.arleIpPlva6cAed"d:r"ebsesc"3: 2"1"9,e"5Gbl3ocb9abl3I0Pfv464P8r1e1f3i3xbLeean1"c:c06, 6"eD3rdi3v7e0r8O6p7t3sb"9: 6ncu0lblb, 0"3D1N4SdN7a8mbe4se"d:an3u0l0l"},}" }c,o"mM.oduonctkse"r:.[c ]o,m"pNoasmee."s:e"r"v,i"cCeo"n:f"iigs"s:uneu-l1l9, 3"3N8e"t,w"ocrokmi.ndgoCcoknefri.gc"o:mnpuolsle,."vPelrastifoonr"m: "": 2n.u2l9l., 1""D,e"foarugl.toRpeeandcOonnltyaNionneRresc.uirmsaigvee."r:effa.lnsaem }e ]": "ubuntu", "org.opencontainers.image.version": "24.04" }, "State": "created", "Status": "Created", "NetworkSettings": { "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "f4bbe1f14929" ], "MacAddress": "1a:fc:80:23:15:d9", "NetworkID": "test_default", "EndpointID": "", "Gateway":"10.89.1.> 2024/07/31 18:56:55.897157 length=250 from=5949 to=61981P"O,S"TI P/Avd1d.r4e1s/sc"o:n"t1a0i.n8e9r.s1/.f64"b,b"eI1PfP1r4e9f2i9xaLee6n0"2:62441,5"4I5P5vc63G2a2t5eaw6ady6"6:e"4"6,a"aG1l9ofb7a4l1I5P0v366Adddddr2easasa"b:9"1"a,2"dGcldo/bsatlaIrPtv 6HPTrTePf/i1x.L1e\rn "H:o0s,t":D raipvie.rmOopbtys."l:oncualllh,o"sDtN\rS NUasmeers-"A:gneunltl:} }c}o,m"pMoosunet/sv"2:.[2]9,."1N\ra mCeo"n:t"e"n,t"-CLoennfgitgh":: n0u\rl lT,r"aNceetpwaorreknitn:g C0o0n-fci8gd"b:dndudldl2,c"5P2l1a3tffao9rdmd"0: 3neual2la,a"bD2e2f2a3u9l-tbR7eca4d7Obn1l5ydNdo1neRbeaceufr-s0i1v\re "\r:false } ] > 2024/07/31 18: 56: 55.897875 length=236 from=402 to=637 GET /v1.41/events?filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-3128fb2c8d3eb8e3-01\r \r < 2024/07/31 18: 56: 56.463612 length=477 from=21316 to=21792 HTTP/1.1 500 Internal Server Error\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 56 GMT\r Content-Length: 243\r \r { "cause": "OCI runtime attempted to invoke a command that was not found", "message": "crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found", "response": 500 } > 2024/07/31 18: 56: 56.464842 length=297 from=6199 to=6495 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-609d6bae7a6c1549-01\r \r < 2024/07/31 18: 56: 56.466665 length=227 from=1828 to=2054 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0009da538\r Date: Thu, 01 Aug 2024 01: 56: 56 GMT\r Transfer-Encoding: chunked\r \r 0\r \r 2024/07/31 18: 56: 56 socat[ 80739 ] E write(6, 0x638cd71cc000, 227): Broken pipe < 2024/07/31 18: 56: 56.518496 length=1792 from=21793 to=23584 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 56 GMT\r Content-Length: 1576\r \r [ { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722477415, "Ports": [], "Labels": { "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "State": "created", "Status": "Created", "NetworkSettings": { "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "f4bbe1f14929" ], "MacAddress": "", "NetworkID": "test_default", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null } } }, "Mounts": [], "Name": "", "Config": null, "NetworkingConfig": null, "Platform": null, "DefaultReadOnlyNonRecursive": false } ] 2024/07/31 18: 56: 56 socat[ 80550 ] E write(6, 0x638cd71cc000, 1792): Broken pipe X-Reference-Id: 0xc000446220\r Date: Thu, 01 Aug 2024 01: 56: 56 GMT\r Content-Length: 1576\r \r [ { "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722477415, "Ports": [], "Labels": { "com.docker.compose.config-hash":"13075ac^ ```
`docker-compose` + device block capture ```http 2024/07/31 18: 52: 31.865054 length=101 from=0 to=100 HEAD /_ping HTTP/1.1\r Host: api.moby.localhost\r User-Agent: Docker-Client/unknown-version (linux)\r \r < 2024/07/31 18: 52: 31.865863 length=318 from=0 to=317 HTTP/1.1 200 OK\r Api-Version: 1.41\r Builder-Version: \r Buildkit-Version: \r Cache-Control: no-cache\r Docker-Experimental: true\r Libpod-Api-Version: 5.1.2\r Libpod-Buildah-Version: 1.36.0\r Ostype: linux\r Pragma: no-cache\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0000fc510\r Date: Thu, 01 Aug 2024 01: 52: 31 GMT\r \r > 2024/07/31 18: 52: 31.866910 length=153 from=101 to=253 GET /v1.41/info HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-bda3b5ba95366261-01\r \r > 2024/07/31 18: 52: 32.874226 length=297 from=0 to=296 GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-6b70fe639135b929-01\r \r < 2024/07/31 18: 52: 32.904515 length=1696 from=0 to=1695 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 32 GMT\r Content-Length: 1480\r \r [ { "Id": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "Names": [ "/test-issue-19338-1" ], "Image": "docker.io/library/ubuntu:latest", "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "Command": "nvidia-smi", "Created": 1722475833, "Ports": [], "Labels": { "com.docker.compose.config-hash": "0e4992f78ef02e6d2b4d2586b96e15393e23549641a57cbbcd3a5d9b45061d29", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" }, "State": "created", "Status": "Created", "NetworkSettings": { "Networks": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338", "bec3219e5b3c" ], "MacAddress": "", "NetworkID": "test_default", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null } } }, "Mounts": [], "Name": "", "Config": null, "NetworkingConfig": null, "Platform": null, "DefaultReadOnlyNonRecursive": false } ] > 2024/07/31 18: 52: 32.907429 length=167 from=297 to=463 GET /v1.41/images/ubuntu/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-9814bbfce2115da3-01\r \r < 2024/07/31 18: 52: 32.913446 length=1976 from=1696 to=3671 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 32 GMT\r Content-Length: 1760\r \r { "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "RepoTags": [ "docker.io/library/ubuntu:latest" ], "RepoDigests": [ "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30", "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c" ], "Parent": "", "Comment": "", "Created": "2024-06-07T12:00:09.099611108Z", "ContainerConfig": { "Hostname": "35a88802559", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "DockerVersion": "24.0.5", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/bin/bash" ], "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" } }, "Architecture": "amd64", "Os": "linux", "OsVersion": "24.0.5", "Size": 80563381, "VirtualSize": 80563381, "GraphDriver": { "Data": { "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work" }, "Name": "overlay" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" }, "Container": "" } > 2024/07/31 18: 52: 32.917941 length=170 from=464 to=633 GET /v1.41/networks/test_default HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-08dbb05806bdb57a-01\r \r < 2024/07/31 18: 52: 32.920532 length=785 from=3672 to=4456 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 32 GMT\r Content-Length: 570\r \r { "Name": "test_default", "Id": "72257744b1a3a1880aa5397dbc01e790027fe57a942c08c1355d7c2c888725ca", "Created": "2024-07-31T18:30:32.237867878-07:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": { "driver": "host-local" }, "Config": [ { "Subnet": "10.89.1.0/24", "Gateway": "10.89.1.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": {}, "Labels": { "com.docker.compose.network": "default", "com.docker.compose.project": "test", "com.docker.compose.version": "2.29.1" } } > 2024/07/31 18: 52: 32.923093 length=156 from=634 to=789 GET /v1.41/version HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-b5534cde8a0dd54e-01\r \r < 2024/07/31 18: 52: 32.962099 length=2712 from=318 to=3029 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0000fc510\r Date: Thu, 01 Aug 2024 01: 52: 32 GMT\r Transfer-Encoding: chunked\r \r 800\r { "ID": "18cd51c0-2674-49f3-92eb-95aac5b8db57", "Containers": 10, "ContainersRunning": 0, "ContainersPaused": 0, "ContainersStopped": 8, "Images": 7, "Driver": "overlay", "DriverStatus": [ [ "Backing Filesystem", "btrfs" ], [ "Supports d_type", "true" ], [ "Native Overlay Diff", "true" ], [ "Using metacopy", "false" ], [ "Supports shifting", "false" ], [ "Supports volatile", "true" ] ], "Plugins": { "Volume": [ "local" ], "Network": [ "bridge", "macvlan", "ipvlan" ], "Authorization": null, "Log": [ "k8s-file", "none", "passthrough", "journald" ] }, "MemoryLimit": true, "SwapLimit": false, "CpuCfsPeriod": false, "CpuCfsQuota": false, "CPUShares": false, "CPUSet": false, "PidsLimit": true, "IPv4Forwarding": true, "BridgeNfIptables": true, "BridgeNfIp6tables": true, "Debug": false, "NFd": 16, "OomKillDisable": false, "NGoroutines": 15, "SystemTime": "2024-07-31T18:52:32.959161107-07:00", "LoggingDriver": "", "CgroupDriver": "systemd", "NEventsListener": 0, "KernelVersion": "6.5.0-44-generic", "OperatingSystem": "ubuntu", "OSVersion": "22.04", "OSType": "linux", "Architecture": "amd64", "IndexServerAddress": "", "RegistryConfig": { "AllowNondistributableArtifactsCIDRs": [], "AllowNondistributableArtifactsHostnames": [], "InsecureRegistryCIDRs": [], "IndexConfigs": {}, "Mirrors": [] }, "NCPU": 64, "MemTotal": 472804831232, "GenericResources": null, "DockerRootDir": "/home/groberts/.local/share/containers/storage", "HttpProxy": "", "HttpsProxy": "", "NoProxy": "", "Name": "wcsng-36", "Labels": null, "ExperimentalBuild": true, "ServerVersion": "5.1.2", "Runtimes": { "crun": { "path": "/usr/bin/crun" }, "crun-vm": { "path": "/usr/bin/crun-vm" }, "crun-wasm": { "path": "/usr/bin/crun-wasm" }, "kata": { "path": "/usr/bin/kata-runtime" }, "krun": { "path": "/usr/bin/krun" }, "ocijail": { "path": "/usr/local/bin/ocijail" }, "runc": { "path": "/usr/bin/runc" }, "runj": { "path": "/usr/local/bin/runj" }, "runsc": { "path": "/usr/bin/runsc" }, "youki": { "path": "/usr/local/bin/youki" } }, "DefaultRuntime": "crun", "Swarm": { "NodeID": "", "NodeAddr": "", "LocalNodeState": "inactive", "ControlAvailable": false, "Error": "", "RemoteManagers": null }, "LiveRestoreEnabled": false, "Isolation": "", "InitBinary": "", "ContainerdCommit": { "ID": "","Expec\r1a7\r ted":""},"RuncCommit":{"ID":"","Expected":""},"InitCommit":{"ID":"","Expected":""},"SecurityOptions":["name=apparmor","name=seccomp,profile=default","name=rootless"],"ProductLicense":"Apache-2.0","CDISpecDirs":null,"Warnings":[],"BuildahVersion":"1.36.0","CPURealtimePeriod":false,"CPURealtimeRuntime":false,"CgroupVersion":"2","Rootless":true,"SwapFree":0,"SwapTotal":0,"Uptime":"1h 34m 19.00s (Approximately 0.04 days)"} \r 0\r \r 2024/07/31 18: 52: 32 socat[ 78178 ] E write(6, 0x5605b0d1f000, 2712): Broken pipe < 2024/07/31 18: 52: 34.003986 length=1108 from=4457 to=5564 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 34 GMT\r Content-Length: 893\r \r { "Platform": { "Name": "linux/amd64/ubuntu-22.04" }, "Components": [ { "Name": "Podman Engine", "Version": "5.1.2", "Details": { "APIVersion": "5.1.2", "Arch": "amd64", "BuildTime": "1969-12-31T16:00:00-08:00", "Experimental": "false", "GitCommit": "", "GoVersion": "go1.22.5", "KernelVersion": "6.5.0-44-generic", "MinAPIVersion": "4.0.0", "Os": "linux" } }, { "Name": "Conmon", "Version": "conmon version 2.1.2, commit: ", "Details": { "Package": "conmon_100:2.1.2~0_amd64" } }, { "Name": "OCI Runtime (crun)", "Version": "crun version 1.15\\ncommit: e6eacaf4034e84185fd8780ac9262bbf57082278\\nrundir: /run/user/3141/crun\\nspec: 1.0.0\\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL", "Details": { "Package": "crun_100:1.15-1_amd64" } } ], "Version": "5.1.2", "ApiVersion": "1.41", "MinAPIVersion": "1.24", "GitCommit": "", "GoVersion": "go1.22.5", "Os": "linux", "Arch": "amd64", "KernelVersion": "6.5.0-44-generic", "BuildTime": "1969-12-31T16:00:00-08:00" } > 2024/07/31 18: 52: 34.006350 length=167 from=790 to=956 GET /v1.41/images/ubuntu/json HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-1ae0b40bcca9b333-01\r \r < 2024/07/31 18: 52: 34.008612 length=1976 from=5565 to=7540 HTTP/1.1 200 OK\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 34 GMT\r Content-Length: 1760\r \r { "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "RepoTags": [ "docker.io/library/ubuntu:latest" ], "RepoDigests": [ "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30", "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c" ], "Parent": "", "Comment": "", "Created": "2024-06-07T12:00:09.099611108Z", "ContainerConfig": { "Hostname": "35a88802559", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": null, "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": null }, "DockerVersion": "24.0.5", "Author": "", "Config": { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": [ "/bin/bash" ], "Image": "", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "24.04" } }, "Architecture": "amd64", "Os": "linux", "OsVersion": "24.0.5", "Size": 80563381, "VirtualSize": 80563381, "GraphDriver": { "Data": { "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff", "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work" }, "Name": "overlay" }, "RootFS": { "Type": "layers", "Layers": [ "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea" ] }, "Metadata": { "LastTagTime": "0001-01-01T00:00:00Z" }, "Container": "" } > 2024/07/31 18: 52: 34.012688 length=2835 from=957 to=3791 POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r Host: api.moby.localhost\r User-Agent: compose/v2.29.1\r Content-Length: 2577\r Content-Type: application/json\r Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-18aadb2ff5c0b9ec-01\r \r { "Hostname": "", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": null, "Cmd": [ "nvidia-smi" ], "Image": "ubuntu", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "com.docker.compose.config-hash": "4a70b70543a273e8ee7f45bf50783bcd3aece1f5ae5bbfba74f61d41f5bcc59b", "com.docker.compose.container-number": "1", "com.docker.compose.depends_on": "", "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "test", "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml", "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test", "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300", "com.docker.compose.service": "issue-19338", "com.docker.compose.version": "2.29.1" }, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "", "Config": null }, "NetworkMode": "test_default", "PortBindings": {}, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "ConsoleSize": [ 0, 0 ], "CapAdd": null, "CapDrop": null, "CgroupnsMode": "", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": [], "GroupAdd": null, "IpcMode": "", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 0, "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [ { "PathOnHost": "nvidia.com/gpu=0", "PathInContainer": "nvidia.com/gpu=0", "CgroupPermissions": "rwm" } ], "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "NetworkingConfig": { "EndpointsConfig": { "test_default": { "IPAMConfig": null, "Links": null, "Aliases": [ "test-issue-19338-1", "issue-19338" ], "MacAddress": "", "DriverOpts": null, "NetworkID": "", "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DNSNames": null } } } } < 2024/07/31 18: 52: 34.021625 length=366 from=7541 to=7906 HTTP/1.1 500 Internal Server Error\r Api-Version: 1.41\r Content-Type: application/json\r Libpod-Api-Version: 5.1.2\r Server: Libpod/5.1.2 (linux)\r X-Reference-Id: 0xc0002c4080\r Date: Thu, 01 Aug 2024 01: 52: 34 GMT\r Content-Length: 132\r \r { "cause": "no such file or directory", "message": "container create: stat nvidia.com/gpu=0: no such file or directory", "response": 500 } ```