search

basecamp / kamal

Deploy web apps anywhere.
https://kamal-deploy.org
MIT License
9.39k stars 359 forks source link

Docker Error on local machine: no space left on device #765

Closed abratashov closed 2 months ago

abratashov commented 2 months ago

After the deployment of several projects with Kamal, I've been stuck on my local machine with the error no space left on device:

kamal setup -d stage
...
#39 pushing layers
#39 pushing layers 0.4s done
#39 ERROR: failed to push registry.gitlab.com/myuser/my-project/my-project-stage:f09ead605afe2604b1e529a97994f90c7eec39f9_uncommitted_82ad302c95ac8ff8: write /home/myuser/.docker/.token_seed: no space left on device
------
 > exporting to image:
------
ERROR: failed to solve: failed to push registry.gitlab.com/myuser/my-project/my-project-stage:f09ead605afe2604b1e529a97994f90c7eec39f9_uncommitted_82ad302c95ac8ff8: write /home/myuser/.docker/.token_seed: no space left on device

On my local machine, only 75MB of free space was left:

user@local_machine$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  468G  444G   75M 100% /

After exploring disk capacity, I've found a lot of ~40GB volumes of projects that I'd deployed before:

sudo du -h --max-depth=1 /var/lib/docker/volumes | sort -h
...
5,0G  /var/lib/docker/volumes/buildx_buildkit_kamal-kamal-blog-staging-multiarch0_state
6,8G  /var/lib/docker/volumes/buildx_buildkit_kamal-kamal-blog-stage-multiarch0_state
23G /var/lib/docker/volumes/buildx_buildkit_kamal-my-project-stage-multiarch0_state
37G /var/lib/docker/volumes/buildx_buildkit_kamal-kamal-blog-multiarch0_state
44G /var/lib/docker/volumes/buildx_buildkit_kamal-my-project-demo-multiarch0_state
45G /var/lib/docker/volumes/buildx_buildkit_kamal-my-project-demo-multiarch0_state
160G  /var/lib/docker/volumes

As I see, the cause is this open Docker issue https://github.com/docker/for-linux/issues/1423 I've found a workaround in this comment https://github.com/docker/for-linux/issues/1423#issuecomment-1958990802

Removing folders freed up space, but it corrupts the build's registry and Docker has to be fully re-installed! :-1:

After the painful reinstalling Docker, I've found in the docs the correct way to remove the builds:

kamal build remove -d stage

Also, I've disabled multiarch building that enormously speeds up the time of image re-building, because my local machine and the remote host have the same AMD64 architecture (this nice feature was added in the version 1.4.0):

# https://kamal-deploy.org/docs/configuration/builders/#using-native-builder-when-multi-arch-isnt-needed
builder:
  multiarch: false

So, I'm closing this ticket!

OS: Ubuntu 22.04.4 LTS Docker: 24.0.5, build 24.0.5-0ubuntu1~22.04.1 Kamal: 1.4.0