Error when running container with `--docker-run`

[asymmetric]

Describe the bug Running a container with devenv container shell --docker-run results in this error:

❯ devenv container shell --docker-run
/nix/store/xg1myaxsmbsls4ldjn0rfaqlzs53l0gb-image-shell.json

Copying container /nix/store/xg1myaxsmbsls4ldjn0rfaqlzs53l0gb-image-shell.json to docker-daemon:shell:latest

Getting image source signatures
Copying blob 61f5dc03ea9d done  
FATA[0005] writing blob: io: read/write on closed pipe  

I've tried with both actual Docker, and with podman, via virtualisation.podman.dockerCompat.

To reproduce

• https://gist.github.com/asymmetric/bc72962c8a15ac7b48c3847b0d57dfeb
• devenv container shell --docker-run

Version

0.6.2

For completeness, here are the outputs of podman and Docker info

podman info

``` host: arch: amd64 buildahVersion: 1.30.0 cgroupControllers: - cpu - io - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: Unknown path: /nix/store/abb2r1159z1xgmyvjkkax20ys5413pzj-conmon-2.1.7/bin/conmon version: 'conmon version 2.1.7, commit: ' cpuUtilization: idlePercent: 22.52 systemPercent: 12.21 userPercent: 65.27 cpus: 4 databaseBackend: boltdb distribution: codename: stoat distribution: nixos version: "23.05" eventLogger: journald hostname: tachikoma idMappings: gidmap: - container_id: 0 host_id: 100 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 6.1.31 linkmode: dynamic logDriver: journald memFree: 511832064 memTotal: 20674895872 networkBackend: netavark ociRuntime: name: crun package: Unknown path: /nix/store/wx7zm8pxwlaibd8719x3izr2g2g936q4-crun-1.8.4/bin/crun version: |- crun version 1.8.4 commit: 1.8.4 rundir: /run/user/1000/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: exists: true path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: "" selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /nix/store/lqybnfwzpmwns3qgmym930m2f55v3fp3-slirp4netns-1.2.0/bin/slirp4netns package: Unknown version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.4 swapFree: 4129390592 swapTotal: 4161794048 uptime: 27h 51m 37.00s (Approximately 1.12 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - docker.io - quay.io store: configFile: /home/asymmetric/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /home/asymmetric/.local/share/containers/storage graphRootAllocated: 283704819712 graphRootUsed: 199158648832 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 0 runRoot: /run/user/1000/containers transientStore: false volumePath: /home/asymmetric/.local/share/containers/storage/volumes version: APIVersion: 4.5.0 Built: 315532800 BuiltTime: Tue Jan 1 01:00:00 1980 GitCommit: "" GoVersion: go1.20.4 Os: linux OsArch: linux/amd64 Version: 4.5.0 ```

docker info

``` Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.10.4) compose: Docker Compose (Docker Inc., 2.18.1) Server: ERROR: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": dial unix /var/run/docker.sock: connect: permission denied errors pretty printing info ```

sudo docker info

``` Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.10.4) compose: Docker Compose (Docker Inc., 2.18.1) Server: Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 0 Server Version: 20.10.23 Storage Driver: btrfs Build Version: Btrfs v6.3 Library Version: 102 Logging Driver: journald Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: v1.7.1 runc version: init version: Security Options: seccomp Profile: default cgroupns Kernel Version: 6.1.31 Operating System: NixOS 23.05 (Stoat) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 19.25GiB Name: tachikoma ID: WGOJ:MUMY:QHOX:HOQ7:XCMI:QZ26:M5KH:VWQI:CXWP:JR3Y:NWMK:QSXW Docker Root Dir: /var/lib/docker Debug Mode: false Username: asymmetric Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true ```

[domenkozar]

Something is wrong with your environment, I cannot reproduce:

devenv container shell --docker-run
/nix/store/v4am5wmdmh37hhka7sxdclqkms18jmf0-image-shell.json

Copying container /nix/store/v4am5wmdmh37hhka7sxdclqkms18jmf0-image-shell.json to docker-daemon:shell:latest

Getting image source signatures
Copying blob 82ba799c9ce5 done  
Copying config 86710afef3 done  
Writing manifest to image destination
Storing signatures
hello from devenv
git version 2.41.0

[asymmetric]

Shouldn't the store paths of the container be the same for both of us?

In fact, using only the files in the gist, I get this hash:

❯ devenv container shell --docker-run
warning: AWS error uploading 'nix-cache-info': Access Denied
/nix/store/jx2052a8c7s7l5kznlym6ckm99srwysb-image-shell.json

Copying container /nix/store/jx2052a8c7s7l5kznlym6ckm99srwysb-image-shell.json to docker-daemon:shell:latest

Getting image source signatures
Copying blob ecf1c6cf64fc done  
FATA[0007] writing blob: io: read/write on closed pipe  

[asymmetric]

The error seems to come from here btw.

[nlewo]

@asymmetric Maybe you could find some useful information in the docker daemon log file.

[nevesenin]

Hi, I ran into the same error while trying out the container feature. I'm currently looking into the different docker setups. I have a rootless docker setup and I'd like to figure out if that is causing the error. In rootless mode docker is not running on unix:///var/run/docker.sock and the provided docker info indicates, that this is the case at your podman setup too @asymmetric. Is that correct, or am I on the wrong track? Regarding useful information from the logs, there is nothing at all when running devenv container shell --docker-run.

[nlewo]

In order to isolate the issue, could you try to copy an image with the upstream Skopeo? The goal is to determine if the issue comes from nix2container or Skopeo.

Something such as

docker save an-image -o /tmp/image.tgz
nix run nixpkgs#skopeo -- copy docker-archive:///tmp/image.tgz docker-daemon://image:latest

[nevesenin]

Ok, I did this:

bsh ❯ docker save postgres -o /tmp/image.tgz
bsh ❯ nix run nixpkgs#skopeo -- --insecure-policy copy docker-archive:///tmp/image.tgz docker-daemon:postgres:latest
Getting image source signatures
Copying blob ed7b0ef3bf5b [--------------------------------------] 8.0b / 80.1MiB
Copying blob 99325206967d [--------------------------------------] 8.0b / 9.7MiB
Copying blob e93e768aace3 [--------------------------------------] 8.0b / 332.0KiB
Copying blob c9a82fac1adb [--------------------------------------] 8.0b / 4.1MiB
Copying blob 6e5fa503410f [--------------------------------------] 8.0b / 24.5MiB
Copying blob e80fccfecee3 [--------------------------------------] 8.0b / 3.4MiB
Copying blob ba93e206464d [--------------------------------------] 8.0b / 2.0KiB
Copying blob 4a99f3f88418 [--------------------------------------] 8.0b / 8.5KiB
Copying blob 32b578a55517 [--------------------------------------] 8.0b / 241.5MiB
Copying blob 00d879a44092 [--------------------------------------] 8.0b / 63.0KiB
Copying blob 75e548dd0a6a [--------------------------------------] 8.0b / 2.0KiB
Copying blob eb7548612644 [--------------------------------------] 8.0b / 3.5KiB
Copying blob 86400f1f9fee [--------------------------------------] 8.0b / 15.5KiB
FATA[0000] writing blob: io: read/write on closed pipe

Didn't work. But I found some interesting flags for skopeo copy like --dest-daemon-host. As you can see, it works with that:

bsh ❯ nix run nixpkgs#skopeo -- --insecure-policy copy --dest-daemon-host unix:///run/user/1000/docker.sock docker-archive:///tmp/image.tgz docker-daemon:postgres:latest
Getting image source signatures
Copying blob ed7b0ef3bf5b done  
Copying blob 99325206967d done  
Copying blob e93e768aace3 done  
Copying blob c9a82fac1adb done  
Copying blob 6e5fa503410f done  
Copying blob e80fccfecee3 done  
Copying blob ba93e206464d done  
Copying blob 4a99f3f88418 done  
Copying blob 32b578a55517 done  
Copying blob 00d879a44092 done  
Copying blob 75e548dd0a6a done  
Copying blob eb7548612644 done  
Copying blob 86400f1f9fee done  
Copying config ab3945c8cf done  
Writing manifest to image destination

Also I found out, that these flags are passed to skopeo by adding them to the devenv.nix like:

containers.shell.defaultCopyArgs = [
  "--dest-daemon-host=unix:///run/user/1000/docker.sock"
];

So, no issue in any of nix2container or skopeo.

[domenkozar]

How does docker itself figure out where's the socket?

[nevesenin]

I tried to find out, but didn't yet.

[nlewo]

@nevesenin is the DOCKER_HOST env variable set?

This could be the way Docker gets the socket path and this is not supported by Skopeo: https://github.com/containers/skopeo/issues/557

[nevesenin]

Sorry for being late. No, DOCKER_HOST is not set.