`devenv container run processes` fails with

[jab]

Describe the bug

I just followed the https://devenv.sh/containers/ docs to try to build and run a container for (essentially a "hello world") Python webapp using devenv for the first time. After picking up the fix for #1083, I can now build a container, but trying to run it fails with an I/O error during the copy step, and doesn't tell me what to do to fix it.

❯ devenv container run processes
• Building processes container ...
• Failed to get cachix caches due to evaluation error
/nix/store/879ki17gj9ki3im0bmlrlmip01xl9a53-image-processes.json
✔ Building processes container in 44.9s.
• Copying processes container ...• Failed to get cachix caches due to evaluation error
• Running /nix/store/7xni5x8sln4nhrbkx4lhc60k5n87yxj4-copy-container /nix/store/879ki17gj9ki3im0bmlrlmip01xl9a53-image-processes.json docker-daemon:

Copying container /nix/store/879ki17gj9ki3im0bmlrlmip01xl9a53-image-processes.json to docker-daemon:processes:latest

Getting image source signatures
Copying blob eacd0e38112a done   |
Copying blob a73ae743743e done   |
FATA[0196] writing blob: io: read/write on closed pipe
✔ Copying processes container in 306.9s.
Error:   × Failed to copy container

I checked and it's not a matter of running out of space on the disk:

❯ df -h
Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  398M     0  398M   0% /dev
tmpfs                     3.9G     0  3.9G   0% /dev/shm
tmpfs                     2.0G  3.6M  2.0G   1% /run
tmpfs                     3.9G  960K  3.9G   1% /run/wrappers
/dev/disk/by-label/nixos   25G   17G  7.2G  70% /
tmpfs                     795M  4.0K  795M   1% /run/user/0

To reproduce

The repo I reproduced this in is here: https://github.com/jab/hello-nix-image-py-webapp

I reproduced this in an x86_64 NixOS environment running recently-updated nixos-unstable:

❯ uname -a
Linux jabdev 6.6.22 #1-NixOS SMP PREEMPT_DYNAMIC Fri Mar 15 18:25:07 UTC 2024 x86_64 GNU/Linux

Version

devenv 1.0.1 (x86_64-linux)

[domenkozar]

@nlewo any ideas?

[nlewo]

It could be because of

tmpfs                     795M  4.0K  795M   1% /run/user/0

@jab Could you try by setting the TMPDIR env variable to /tmp before running devenv container run processes (i don't know if devenv modifies this env variable)?

If this doesn't help, it would be nice to run some commands with the upstream Skopeo to know if this issue is related to nix2container or Skopeo. For instance, something such as:

nix run nixpkgs#skopeo -- copy docker://alpine docker-daemon://alpine

[jab]

@nlewo Same I/O error after setting TMPDIR:

hello-nix-image-py-webapp on  main is 📦 v0.1 via 🐍 v3.11.8 via ❄️  impure (devenv-shell-env) on ☁️  (us-east-1)
❯ echo $TMPDIR
/tmp

hello-nix-image-py-webapp on  main is 📦 v0.1 via 🐍 v3.11.8 via ❄️  impure (devenv-shell-env) on ☁️  (us-east-1)
❯ devenv container run processes
• Building processes container ...
• Failed to get cachix caches due to evaluation error
/nix/store/h4jb5kr247b6jhpkd4pj8yrinprmlf48-image-processes.json
✔ Building processes container in 16.4s.
• Copying processes container ...• Failed to get cachix caches due to evaluation error
• Running /nix/store/7xni5x8sln4nhrbkx4lhc60k5n87yxj4-copy-container /nix/store/h4jb5kr247b6jhpkd4pj8yrinprmlf48-image-processes.json docker-daemon:

Copying container /nix/store/h4jb5kr247b6jhpkd4pj8yrinprmlf48-image-processes.json to docker-daemon:processes:latest

Getting image source signatures
Copying blob eacd0e38112a done   |
Copying blob 5a199d0ce155 done   |
FATA[0224] writing blob: io: read/write on closed pipe
✔ Copying processes container in 227.7s.
Error:   × Failed to copy container

As for further testing, I wonder if you might be able to try to reproduce this yourself so you could have a faster feedback loop as you try various things to debug this. (I specifically waited till devenv reached 1.0 before I gave it another try since I'm unfortunately a bit too short on time right now to spend a lot of time debugging stuff like this -- wish I could help more right now!)

[nlewo]

wow, the size of a layer is 6GB: it takes few minutes on my laptop to be sent to the Docker deamon but it starts successfully:

devenv container run processes
• Building processes container ...
• Using Cachix: devenv
/nix/store/8nbp6bmwbna3vcs0wvyf0k0jkrb9llrc-image-processes.json
✔ Building processes container in 20.2s.
• Running /nix/store/7xni5x8sln4nhrbkx4lhc60k5n87yxj4-copy-container /nix/store/8nbp6bmwbna3vcs0wvyf0k0jkrb9llrc-image-processes.json docker-daemon: 

Copying container /nix/store/8nbp6bmwbna3vcs0wvyf0k0jkrb9llrc-image-processes.json to docker-daemon:processes:latest

Getting image source signatures
Copying blob eacd0e38112a done   | 
Copying blob 75d94c1b557f done   | 
Copying config 60a5a6cc0f done   | 
Writing manifest to image destination
✔ Copying processes container in 483.9s.

^[2WARNING: pre-commit-hooks.nix: git command not found; skipping installation.
{"level":"warn","error":"open /env/.config/process-compose/settings.yaml: no such file or directory","time":"2024-04-02T20:30:06Z","message":"Error reading settings file /env/.config/process-compose/settings.yaml"}
✔ Running processes container in 21.5s.

So, i'm pretty sure it is an issue related to the storage available on your machine...

If you have the SSH to the machine, it would be nice to df -h during the image copy to see if your storage available space decrease.

[jab]

(Haven't had time yet to log back into the machine but) some quick thoughts in the meantime:

1. My df -h output above indicated 7.2G free on /. Does it make sense then that this would not be enough to copy a 6G image (e.g. is 2x the size of the image required)?

2. Are you checking the current amount of space available before attempting the copy to make sure there is enough, and if not, erroring early? ("Error: This operation requires XYZ available space but only ABC is available.")

3. Why is a devenv.nix that basically just enables Python along with a "hello world" Python module (note: no dependencies other than the Python standard library) resulting in such an enormous container image?