search

ufoym / deepo

Setup and customize deep learning environment in seconds.
http://ufoym.com/deepo
MIT License
6.32k stars 749 forks source link

Pull not working - No space left on device #108

Closed ghost closed 5 years ago

ghost commented 5 years ago

Hey,

when trying to pull the image I get the following output.

$ docker pull ufoym/deepo
Using default tag: latest
latest: Pulling from ufoym/deepo
898c46f3b1a1: Pull complete 
63366dfa0a50: Pull complete 
041d4cd74a92: Pull complete 
6e1bee0f8701: Pull complete 
112097260ef3: Pull complete 
30a67c795176: Pull complete 
0d286a7b6e12: Pull complete 
5d089cae17f6: Pull complete 
55dd29113cda: Pull complete 
987b8ae28a17: Pull complete 
0395fffb1efc: Extracting  2.904GB/2.904GB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: mkdir /usr/local/lib/python3.6/dist-packages/torch/testing: no space left on device

Someone suggested checking my inode space

$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
dev             2036728    561  2036167    1% /dev
run             2039268    902  2038366    1% /run
/dev/dm-0      14491648 486294 14005354    4% /
tmpfs           2039268    170  2039098    1% /dev/shm
tmpfs           2039268     18  2039250    1% /sys/fs/cgroup
tmpfs           2039268    168  2039100    1% /tmp
/dev/sda1             0      0        0     - /boot/efi
tmpfs           2039268     31  2039237    1% /run/user/1000

I still have 170GB of free disk space, so that shouldn't be an issue. I've already removed all my unused images to no avail.

Any suggestions?

ufoym commented 5 years ago

See https://github.com/Kaggle/docker-python/issues/19

ghost commented 5 years ago

The solution mentioned in that issue didn't work out for me, however i found a solution that worked for me. I'll describe it for anyone facing the same problem.

Some relevant lines from docker info:

Server Version: 18.09.5-ce
Kernel Version: 4.19.36-1-MANJARO
Operating System: Manjaro Linux

I used the default install of docker from the Manjaro repo. For some reason this was configured to use xfs as the "Backing Filesystem" for the Docker Storage Driver. This meant it wasn't able to extract this image fully, due to lack of storage space.

The fix for me was adding: /etc/docker/daemon.json

{
    "storage-driver": "overlay2"
}

Restart docker afterwards. This changed the Backing Filesystem to the actual host machine's FS (ext4).