Open nelsonic opened 6 years ago
still getting:
remote: Error response from daemon: Error processing tar file(exit status 1): write /app/node_modules/puppeteer/.local-chromium/linux-575458/chrome-linux/chrome: no space left on device
https://stackoverflow.com/questions/24671621/no-space-left-on-device
Such difference between the output of du -sh and df -h may happen if some large file has been deleted, but is still opened by some process. Check with the command lsof | grep deleted to see which processes have opened descriptors to deleted files. You can restart the process and the space will be freed.
Digital Ocean dashboard indicates that we are out of SSD and RAM space ... π
so I logged into the machine and ran:
reboot -h now
Based on: https://serverfault.com/questions/620595/best-way-to-gracefully-restart-centos Of the system RAM 1882872 (1.8Gb) total, 1299252 (1.29Gb) is used and only 268800 (268Mb) available.
That did not have the desired effect because there are still way too many running Docker Containers (52!)
So I decided to "Nuke" the droplet with a power cycle! (which is obviously not recommended!)
Obviously now there is no docker process running:
[root@centos-dokku-paas ~]# docker ps -a -q
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
After the power cycle:
[root@centos-dokku-paas ~]# sudo systemctl start docker
[root@centos-dokku-paas ~]# docker -D info
Containers: 52
Running: 52
Paused: 0
Stopped: 0
Images: 270
Server Version: 18.05.0-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.2.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.796GiB
Name: centos-dokku-paas
I read: https://medium.com/the-code-review/clean-out-your-docker-images-containers-and-volumes-with-single-commands-b8e38253c271 and ran the command:
docker system prune --all --force --volumes
Which had the following output:
Deleted Containers:
5850620d78b98512276bd03816cd61c70eefd5c56f64a513edfc713188fa16f2
152487e3869812f40fb48f26a667d75959b950650c5a5fa257999a8561bbd081
61a40b85e3538dccf5bc5ad040f751fe1d0058db5a9e760fc2629aa680e3f847
0ec7301b697a24e758fe8f0193d2c2233008779f5ee49e652f95b5603c70b95b
dca74ebdb1f74a233c6fc6c8bfa9d080294f65f4bf6fea11955ea95b19bfbfce
755b3c5eca7fdba45bab39b9e05e4d9423cea9a8dea7444f4cc670161a9068c1
Deleted Images:
untagged: dokku/ci-master:latest
deleted: sha256:fce0a33a7768bc4bfa73ad324bc9ac546eb6bafad1a725e9c9118febbe8f556b
deleted: sha256:9efa1313d1bd4e12c2427c5715cd57db63d7a5236709f43a8c4938be9a634843
deleted: sha256:397980186cb30c1845f2b32f17f144723da417c135bc081d38c79cde5579617c
deleted: sha256:085bf52ee253902a5d7198143684d7138a7bad77052cbc1c05675e5765e18b38
deleted: sha256:26e4278b79a7ee649e9433fda844827e19b31eb3063c3d7890106dd3400b3d0e
deleted: sha256:f3efa2da50afd418fac27263a2a527c7ab46a2fb5655ee80080a5ca2237adfd4
untagged: dokku/ci-ui-tweaks:latest
deleted: sha256:614ec0e1e26a37d87f2611dbd3d44d7c37d265aff8c401bb2b87f33a7aca4042
deleted: sha256:11fb7656bb030b4ccb582540781e79994dbefdc24c2407713be18f613ea2d803
deleted: sha256:bfdeaf35596a52fc922050789fb0a0cba566ce3455e1e9050eeab020fe00eff6
deleted: sha256:3edbce420a30acb007e69490b303684ae7b36bc78b358d9d1dc05ad92356c38c
deleted: sha256:70b2518d2f72d6575845cf0159c97e9511e4163805a56528ad2bd7c64c7c5b00
deleted: sha256:c43535bc88aac739b558d13120c98cc2d59b8fe783c5ca2a73a88500f5bda6ac
untagged: dokku/ci-calendar-pill-yachts-page:latest
deleted: sha256:8f5759fb0d9ce975f6178ee2f39deb521f8a04808b2bd1b23e7c8b4885d8d568
deleted: sha256:4e60f958cb8856ed78a537434ff96b0d942caa937caf686104c78c5587c2defc
deleted: sha256:42fc1067a8571f92f2045d51aa5d742ff019f7624bcdc49df827f7b8cae7354c
deleted: sha256:b759b2257fd1537cb12273ee882b77b43900d50edf62b99571a5c08dd327015f
deleted: sha256:776b61880cd5cf6d77dca1f2d561d086a33e3d348569cf8b6059a6408273b60b
deleted: sha256:ecff6fcb5655d725da328a8041ead281bf236daec317142a66b6b544b39d7fb4
untagged: gliderlabs/herokuish:v0.4.2
untagged: gliderlabs/herokuish@sha256:32cbd10ca939e4fbfba2025a8d554b364f114af87531dc487ca25f4cab92a84e
Total reclaimed space: 3.356GB
Which is a start ... the RAM and Disk are no longer at 100% ...
But it's still not a "solution"!
Re-started the build for the app that we are testing ...
Now we have 420Mb of available RAM:
But we still have way too many running docker containers:
A complete Docker system clean can now be achieved by linking this with the full prune command covered earlier in this post.
docker container stop $(docker container ls -a -q) && docker system prune -a -f --volumes
Now when we run docker ps
there are zero running docker containers.
let's rebuild all the dokku
containers: dokku ps:rebuildall
Now the CPU, RAM and Disk are back to "normal" levels:
When attempting to auto-deploy from CircleCI we are seeing the following error:
Full Stack Trace: