search

PharmaLedger-IMI / fgt-workspace

UC3 Finished Goods Traceability
MIT License
5 stars 3 forks source link

fgt-tst-pl disk space exhausted #124

Open joaoluis-pdm opened 2 years ago

joaoluis-pdm commented 2 years ago

Seems to be the overlay2 docker folder:

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K    /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
59M    /var/lib/docker/image
128K    /var/lib/docker/network
9.8G    /var/lib/docker/overlay2
16K    /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
...

Does not show up on docker system df... :-(

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         0         7.232GB   7.232GB (100%)
Containers      0         0         0B        0B
Local Volumes   16        0         11.16GB   11.16GB (100%)
Build Cache     0         0         0B        0B

$ docker system df -v
Images space usage:
REPOSITORY                         TAG              IMAGE ID       CREATED        SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
joaoluispdm/pharmaledger-imi-fgt   v0.10.7          40ec353cad55   4 weeks ago    2.412GB   110.6MB       2.301GB       0
api_mah-bayer                      latest           155fcb4d9f8a   5 weeks ago    2.408GB   138.2MB       2.269GB       0
joaoluis-pdm commented 2 years ago

fgt-tst-pl migrated to SSD1, and / FS grown from 60GB to 120GB. Details in VMWare.docx

After reboot

# during a restart of all containers

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
2.3M    /var/lib/docker/containers
59M /var/lib/docker/image
128K    /var/lib/docker/network
26G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
15G /var/lib/docker/volumes

# one hour later

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
45M /var/lib/docker/containers
59M /var/lib/docker/image
128K    /var/lib/docker/network
40G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
15G /var/lib/docker/volumes

root@fgt-tst-pl:~# date
Wed Oct 19 09:06:03 UTC 2022

overlay2 has grown from 9.8GB to 26GB to 40GB... ???!!!

joaoluis-pdm commented 2 years ago

Some tips on overlay2 https://github.com/moby/moby/issues/33775

# du -shc /var/lib/docker/overlay2/*/diff | grep total
12G total

Not helpful. 12GB would be reasonable ... after all these months... But where are the 40GB ?

pharmaledger@fgt-tst-pl:~$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         3         7.232GB   4.78GB (66%)
Containers      17        17        1.706GB   0B (0%)
Local Volumes   16        15        11.32GB   352.3MB (3%)
Build Cache     0         0         0B        0B

Something smells bad over docker...

joaoluis-pdm commented 2 years ago

Detected again on 2022-11-10 10h.

Before any cleanup/restart.

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G  113G     0 100% /

root@fgt-tst-pl:~# du -sh /var/*
1.9M    /var/backups
104M    /var/cache
4.0K    /var/crash
115G    /var/lib
4.0K    /var/local
0   /var/lock
9.4G    /var/log
4.0K    /var/mail
4.0K    /var/opt
0   /var/run
80K /var/snap
28K /var/spool
36K /var/tmp
20K /var/www

/var/lib 115GB ? Docker issues ?

root@fgt-tst-pl:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          6         3         7.232GB   4.78GB (66%)
Containers      17        17        1.706GB   0B (0%)
Local Volumes   16        15        33.14GB   352.3MB (1%)
Build Cache     0         0         0B        0B

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
32G /var/lib/docker/containers
59M /var/lib/docker/image
128K    /var/lib/docker/network
40G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
43G /var/lib/docker/volumes

We seem to be using 40GB overlay2 + 43GB volumes...

Which ones ?

root@fgt-tst-pl:~# du -sh /var/lib/docker/overlay2/*
288M    /var/lib/docker/overlay2/02595d8cdacea04a2f593e64ab4d27793f3647024c2a5c56954afd14297cbbcc
128K    /var/lib/docker/overlay2/0698b709b7c38e30feecb9fe255ba5e08ac42821548c5670f8ce56e0a8556c77
1.4G    /var/lib/docker/overlay2/09648db2c009b1d77699864451bd6d562b0011dd2c2942ccda67b06210010eaa
2.0G    /var/lib/docker/overlay2/0b09896a383901da53cc5d15621f99509151653328114107b8743fb29b0dff8b
40K /var/lib/docker/overlay2/0b09896a383901da53cc5d15621f99509151653328114107b8743fb29b0dff8b-init
2.0G    /var/lib/docker/overlay2/0bc88d7bf9f683072ef3c15d4f2370c0359c8ed48ef7eccd6deb7bf77e4e1ca5
40K /var/lib/docker/overlay2/0bc88d7bf9f683072ef3c15d4f2370c0359c8ed48ef7eccd6deb7bf77e4e1ca5-init
2.0G    /var/lib/docker/overlay2/10b772916195991bb5ef2cb0b1fa051407e83b9a8a64188dd4444da1f22020a8
40K /var/lib/docker/overlay2/10b772916195991bb5ef2cb0b1fa051407e83b9a8a64188dd4444da1f22020a8-init
17M /var/lib/docker/overlay2/13e4001d5362c3254ac1c47432c4ca30af08017b5789aa97325068beb4c49500
103M    /var/lib/docker/overlay2/140aec3bf357b0fe38d5296b2226ab1c30cb75b4ed717212b2800c87631d87d5
6.0M    /var/lib/docker/overlay2/1aaf1364256cad9c49550dd78466069a47ed260dc71c2cbf02b46a78f34e5751
2.0G    /var/lib/docker/overlay2/2367181620da0d3e15fd164e32604b309cfc821074f8407767edf59f6beff087
40K /var/lib/docker/overlay2/2367181620da0d3e15fd164e32604b309cfc821074f8407767edf59f6beff087-init
2.0G    /var/lib/docker/overlay2/2c00280a13676389f03f6bd296e44a218625bd0b64b06fab72607ab605988832
40K /var/lib/docker/overlay2/2c00280a13676389f03f6bd296e44a218625bd0b64b06fab72607ab605988832-init
2.0G    /var/lib/docker/overlay2/2f41649815d60852794e2ab8c7765fb8f60138f4d9d733094a46f1e2ecd8f5af
40K /var/lib/docker/overlay2/2f41649815d60852794e2ab8c7765fb8f60138f4d9d733094a46f1e2ecd8f5af-init
1.6G    /var/lib/docker/overlay2/3121128b8a2f6eecbc8146554cfacc43cbdbdae29870909a86af6bd08e3b3b47
2.0G    /var/lib/docker/overlay2/39ff7d2547c7becc02a7e6f1340677b1fd96c6f88cae86bbcf38f44a8399d2e7
40K /var/lib/docker/overlay2/39ff7d2547c7becc02a7e6f1340677b1fd96c6f88cae86bbcf38f44a8399d2e7-init
56M /var/lib/docker/overlay2/41dfd4c33f8658beced6b0b384a0be5cd211afa1fea701a0ca742e9738b6458d
40K /var/lib/docker/overlay2/41dfd4c33f8658beced6b0b384a0be5cd211afa1fea701a0ca742e9738b6458d-init
18M /var/lib/docker/overlay2/42e6924cf51e722fff586c774024b3d1bb4d7597d5cfdd2a2d077215975c5873
2.0G    /var/lib/docker/overlay2/4526db663a9ae7f48da41858e6194e9aa6eee2db1980fe30286e3bb744135b3d
40K /var/lib/docker/overlay2/4526db663a9ae7f48da41858e6194e9aa6eee2db1980fe30286e3bb744135b3d-init
18M /var/lib/docker/overlay2/4eabf59e3e5682697b1895092c084cbfeb2c3f37dd83d16c6594af40accce6aa
44K /var/lib/docker/overlay2/531e76eefc35f0ac5d37223a952f4fccaf0eb319192c58183c215ed60abe586a
128K    /var/lib/docker/overlay2/55da8263a951fb24d88f89ad1c8b03923945ffb3c0b479df5e1b02bf263b1b2f
36K /var/lib/docker/overlay2/5ced429f99f7cae503b6cb4563c41dcf112705c37e8e75074143b826689dcc11
2.0G    /var/lib/docker/overlay2/63ac33f69a453c23f86a1260e19cec218ab816899051ff07a1ec938ab5fbb41a
40K /var/lib/docker/overlay2/63ac33f69a453c23f86a1260e19cec218ab816899051ff07a1ec938ab5fbb41a-init
101M    /var/lib/docker/overlay2/6686ebc9872e2aa7be00a20ae0bec2b7efd4be9c9fd5f930fb0b75087fa98926
40K /var/lib/docker/overlay2/6686ebc9872e2aa7be00a20ae0bec2b7efd4be9c9fd5f930fb0b75087fa98926-init
24K /var/lib/docker/overlay2/69a393cfabb5fcab6ee49ffbc8d73ea20e7c680880363035e74c408b4316c5d7
50M /var/lib/docker/overlay2/6be604d0a97020b790608d73292b2a02199700b5880eea460fa73b08fa4bf68a
1.5G    /var/lib/docker/overlay2/6e801866ddd860ff3e0cbf80ca8981157474c1507ea821fb8ff0b1907e3b13c6
239M    /var/lib/docker/overlay2/6eaa5c0a3e63efc854cd035d2001335e26e2bca7b2275685f5bf3335a5621d4f
4.7M    /var/lib/docker/overlay2/6eeb8f30c0b90712fdc4fdaeafb85b6d38a9620711cd047820604999365e0af6
24K /var/lib/docker/overlay2/722038ccd1a834384e3bd2f3dff6eed49720bc77c1ba0eee6252075eef2553bd
180K    /var/lib/docker/overlay2/732f00ebe13b66291bff15347812251dce5f0da0db30e0b9d5e69cfd3871c5b7
6.0M    /var/lib/docker/overlay2/73e7447d064f8b518a4d0a9216c0c7adb75750ef0db2fa66cd7ccf5846b73fed
288M    /var/lib/docker/overlay2/8067e32cfbea99ec527a817f2e4a301f0ffa825b8ab0f200a5f7f99f1f1419a7
2.0G    /var/lib/docker/overlay2/85142286c283cff68f3867bf2efe22084f160f9ce509d9ff8e8a6f6eae1b87d2
40K /var/lib/docker/overlay2/85142286c283cff68f3867bf2efe22084f160f9ce509d9ff8e8a6f6eae1b87d2-init
180K    /var/lib/docker/overlay2/87da5674f4e6eadb442d514874a177509708e6e4ec2b272096f6c0d1945afb21
91M /var/lib/docker/overlay2/8fb263ae881882ea60dcdc4304d17bbe94b388b05a5aa29d6e086aa5bdab7192
24K /var/lib/docker/overlay2/a4db6f71dfbba702bccc74664be78067f233adf6005b38305781686dccee5dc5
1.4G    /var/lib/docker/overlay2/a5c3a45eece2b0e23c67e5cda7d233ba926b9dd62fb78f1a2af08afa1bbd1779
1.5G    /var/lib/docker/overlay2/b15de220baab18e7abe375a0851e4e90ede64f432a979847e1652cab486ef24e
2.0G    /var/lib/docker/overlay2/b168e043ea668b15f4dd68104e95a7bf793c96f1c23f9dea54abb055e3a10c60
40K /var/lib/docker/overlay2/b168e043ea668b15f4dd68104e95a7bf793c96f1c23f9dea54abb055e3a10c60-init
128K    /var/lib/docker/overlay2/b17cddb97b0979f7ec1a67420c425add340ec1ce5299e3c1470d0038b143dce6
1.5G    /var/lib/docker/overlay2/bb3e48a5e82e77a11434aa8e13a680bf82c91b93c932695a855ef3bdd49c2b8b
28K /var/lib/docker/overlay2/be0d646daedda0c05a6f48a21f82758a012ded2ec525a2d3cbe34865e0d8aef3
2.0G    /var/lib/docker/overlay2/c22f1bde90bca49ebcda988890b220071855eee8b9423dfe5ee12851398fc88b
40K /var/lib/docker/overlay2/c22f1bde90bca49ebcda988890b220071855eee8b9423dfe5ee12851398fc88b-init
1.9G    /var/lib/docker/overlay2/c35807d874ea3f07e21b40152e48b73a9fdbabb77ad7c23873c9bb4c0463e5e4
40K /var/lib/docker/overlay2/c35807d874ea3f07e21b40152e48b73a9fdbabb77ad7c23873c9bb4c0463e5e4-init
17M /var/lib/docker/overlay2/c8698b13ab76cf5d4b62f51d7a3d7f82b8a77aaeb2bbbfa2e8479d34ba74d154
20K /var/lib/docker/overlay2/d6873530258222f435862d73466a69d97a15a33658e52e1597d10712c4daa6bd
2.0G    /var/lib/docker/overlay2/d97546d294f59fbff03af4aeca122c9c297a4c6ca5e73b1db815c73818ae5a0d
40K /var/lib/docker/overlay2/d97546d294f59fbff03af4aeca122c9c297a4c6ca5e73b1db815c73818ae5a0d-init
24K /var/lib/docker/overlay2/d98648f0cbef0adc270cd21463c1e4ccf80727c60ebdff16e2812e7fd94180fa
2.0G    /var/lib/docker/overlay2/e07d0cb4d7ebf5f9833b6c421c83ab9f617e1aa0e480d39872d2ba07e44edc27
40K /var/lib/docker/overlay2/e07d0cb4d7ebf5f9833b6c421c83ab9f617e1aa0e480d39872d2ba07e44edc27-init
7.6M    /var/lib/docker/overlay2/e0bbad577616111cfcfe21e297c9c51f07834b8bc4605aa57ef2dd174b40b48d
24K /var/lib/docker/overlay2/e3451e73a9ba2a7465775858a9006d357d6d9e47b01715fd41d675f09cb859ef
36K /var/lib/docker/overlay2/e541f2e7fe54cecdea1d036b8e287ff1858bc6912c3c1403aad677b3cfcb72a8
24K /var/lib/docker/overlay2/eb3f43cb7979a9d2675a390b538ee73edca2c5f2eae23aa8e832e2985c084e7e
180K    /var/lib/docker/overlay2/fc4cc08d20fc72f6a0351403aba376fc734c2f849d8d53d316ccb7404e7456b5
28K /var/lib/docker/overlay2/fe760c33ce6ea5dcd4387487bd0774876cb6664929b3d5dc3644df2ab063cf7c
292K    /var/lib/docker/overlay2/l

root@fgt-tst-pl:~# du -sh /var/lib/docker/volumes/*
58M /var/lib/docker/volumes/api_mah-bayer_vol
58M /var/lib/docker/volumes/api_mah-gsk_vol
59M /var/lib/docker/volumes/api_mah-msd_vol
58M /var/lib/docker/volumes/api_mah-novo-nordisk_vol
58M /var/lib/docker/volumes/api_mah-pfizer_vol
58M /var/lib/docker/volumes/api_mah-roche_vol
58M /var/lib/docker/volumes/api_mah-sanofi_vol
58M /var/lib/docker/volumes/api_mah-takeda_vol
58M /var/lib/docker/volumes/api_pha1_vol
59M /var/lib/docker/volumes/api_pha2_vol
42G /var/lib/docker/volumes/api_traceability_vol
58M /var/lib/docker/volumes/api_whs-takeda_vol
129M    /var/lib/docker/volumes/api_whs-zuellig_vol
58M /var/lib/docker/volumes/api_whs1_vol
58M /var/lib/docker/volumes/api_whs2_vol
0   /var/lib/docker/volumes/backingFsBlockDev
409M    /var/lib/docker/volumes/external-volume
32K /var/lib/docker/volumes/metadata.db

Seems to be the traceability domain that is growing...

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/*
4.0K    /var/lib/docker/volumes/api_traceability_vol/_data/channels
32K /var/lib/docker/volumes/api_traceability_vol/_data/config
42G /var/lib/docker/volumes/api_traceability_vol/_data/domains
8.0K    /var/lib/docker/volumes/api_traceability_vol/_data/messaging
4.0K    /var/lib/docker/volumes/api_traceability_vol/_data/mq_JWT_Auth_Seeder
88K /var/lib/docker/volumes/api_traceability_vol/_data/mqs
4.0K    /var/lib/docker/volumes/api_traceability_vol/_data/notifications
4.0K    /var/lib/docker/volumes/api_traceability_vol/_data/read.me

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/domains/*
28M /var/lib/docker/volumes/api_traceability_vol/_data/domains/default
42G /var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability
33M /var/lib/docker/volumes/api_traceability_vol/_data/domains/vault

root@fgt-tst-pl:~# du -sh  /var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/*
501M    /var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/anchors
41G /var/lib/docker/volumes/api_traceability_vol/_data/domains/traceability/brick-storage

Get some free space.

pharmaledger@fgt-tst-pl:~$ ls -lh images/
total 5.1G
-rw-rw-r-- 1 pharmaledger pharmaledger 5.0G Oct 19 09:15 delete-when-space-needed.empty
joaoluis-pdm commented 2 years ago

Logs have been backed to the private sharepoint.

Stopping appers to free 40GB

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ docker-compose down
pharmaledger@fgt-tst-pl:~/logs$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   70G   43G  63% /
pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ ./bootCompose.sh tst
joaoluis-pdm commented 1 year ago

OS upgraded, volumes and images cleared.

pharmaledger@fgt-tst-pl:~$ docker system prune -a
pharmaledger@fgt-tst-pl:~$ docker volume prune

pharmaledger@fgt-tst-pl:~$ dd if=/dev/zero of=~/images/delete5G.when.needed bs=1G count=5
pharmaledger@fgt-tst-pl:~$ dd if=/dev/zero of=~/images/delete5Gb.when.needed bs=1G count=5

Docker re-installed as instructed on https://docs.docker.com/engine/install/ubuntu/#uninstall-docker-engine

joaoluis-pdm commented 1 year ago 
pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /
... apt update/upgrade/autoremove 
root@fgt-tst-pl:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.8G     0  7.8G   0% /dev
tmpfs                              1.6G  1.7M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          974M  209M  698M  24% /boot
/dev/loop5                          56M   56M     0 100% /snap/core18/2785
/dev/loop2                         174M  174M     0 100% /snap/lxd/25112
/dev/loop6                          64M   64M     0 100% /snap/core20/1974
/dev/loop10                         74M   74M     0 100% /snap/core22/858
/dev/loop1                          64M   64M     0 100% /snap/core20/2015
/dev/loop4                          56M   56M     0 100% /snap/core18/2790
/dev/loop8                          41M   41M     0 100% /snap/snapd/19993
/dev/loop3                          74M   74M     0 100% /snap/core22/864
/dev/loop9                         176M  176M     0 100% /snap/lxd/25505
/dev/loop0                          41M   41M     0 100% /snap/snapd/20092
tmpfs                              1.6G     0  1.6G   0% /run/user/1001
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/a6313b137cfdfa7ce99bd375f27603dd96f1e4542bc9f8b295bf502cef9f020e/merged
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/86c11e3484d6800fa00c95d81d9c2109b6db1e7514c0b738c1705085731457ce/merged
overlay                            117G   87G   26G  77% /var/lib/docker/overlay2/df683b97e20eea63a255acf31c7e28bae4c7b449469bf521d02b40f9304a89fb/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722/mounts/shm
root@fgt-tst-pl:~# reboot

pharmaledger@fgt-tst-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   87G   26G  77% /

pharmaledger@fgt-tst-pl:~$ docker logs -f --since=1h fgt-workspace 
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Apparently I am hitting https://github.com/docker/for-linux/issues/140

The root@fgt-tst-pl:~# grep -P '\x00' /var/lib/docker/containers/**/*.log gives no results.

Workaround: Using --tail on commands such as

docker log -f --tail 1000 fgt-workspace

Logs are taking some disk space, but should be less than 6GB.

root@fgt-tst-pl:~# ls -lh /var/lib/docker/containers/**/*.log
-rw-r----- 1 root root  20M Sep 27 07:10 /var/lib/docker/containers/090b06f15dcd0be58fbb05912e1a60e19d9fc16bf9b0ea275d15596b327476c0/090b06f15dcd0be58fbb05912e1a60e19d9fc16bf9b0ea275d15596b327476c0-json.log
-rw-r----- 1 root root 396M Sep 27 09:15 /var/lib/docker/containers/226bf4318049e64615ae03622c51509b7a4ea4a67686e216a7b0ddaf899681fc/226bf4318049e64615ae03622c51509b7a4ea4a67686e216a7b0ddaf899681fc-json.log
-rw-r----- 1 root root  20M Sep 27 07:17 /var/lib/docker/containers/2cb0d8f951408b5ea5b942c07000a51071d5554ce90995de47f260d74392b80a/2cb0d8f951408b5ea5b942c07000a51071d5554ce90995de47f260d74392b80a-json.log
-rw-r----- 1 root root 509M Sep 27 09:15 /var/lib/docker/containers/596d2c2565d07388037ecbe91a1062a2d24a4165edfa177d3c4882b2bcd4fbfe/596d2c2565d07388037ecbe91a1062a2d24a4165edfa177d3c4882b2bcd4fbfe-json.log
-rw-r----- 1 root root  20M Sep 27 07:07 /var/lib/docker/containers/738f9547fd4149bcba4719c32c99def8f5b675293bb240d565f231a4202d6fd9/738f9547fd4149bcba4719c32c99def8f5b675293bb240d565f231a4202d6fd9-json.log
-rw-r----- 1 root root  19M Sep 27 06:55 /var/lib/docker/containers/843d2c8bd41716ecbfad407ee949654066980f8a97934bfdd1df5a23b65bbfc6/843d2c8bd41716ecbfad407ee949654066980f8a97934bfdd1df5a23b65bbfc6-json.log
-rw-r----- 1 root root  35M Sep 27 07:00 /var/lib/docker/containers/868627bbf8a26ab2dabf8dcf74aefdc7c24f004ef2116732789610bcef555ace/868627bbf8a26ab2dabf8dcf74aefdc7c24f004ef2116732789610bcef555ace-json.log
-rw-r----- 1 root root  20M Sep 27 07:10 /var/lib/docker/containers/9455722e74eae91299653cadf62e59cda6145c6d8c52290c5ba5227bcbd36cc3/9455722e74eae91299653cadf62e59cda6145c6d8c52290c5ba5227bcbd36cc3-json.log
-rw-r----- 1 root root  20M Sep 27 07:22 /var/lib/docker/containers/989c4ec69a17a9af11316c46d925eceb5396a5d48923521bc71824e58873b6fb/989c4ec69a17a9af11316c46d925eceb5396a5d48923521bc71824e58873b6fb-json.log
-rw-r----- 1 root root  21M Sep 27 07:00 /var/lib/docker/containers/99e64cead7c65d40eb742a88d1c3a9f9af9730068486fe5e0bb0ff3b7cf48302/99e64cead7c65d40eb742a88d1c3a9f9af9730068486fe5e0bb0ff3b7cf48302-json.log
-rw-r----- 1 root root  32M Sep 27 06:42 /var/lib/docker/containers/b5e9b2d8972a1729f98db4ca64174f3b115e2c502b64730606b1a8f0c1c08466/b5e9b2d8972a1729f98db4ca64174f3b115e2c502b64730606b1a8f0c1c08466-json.log
-rw-r----- 1 root root  20M Sep 27 09:16 /var/lib/docker/containers/c999718ebcf935e46e695eb9a9b37bb9ced668676f975613ee6bbdcbe18663f9/c999718ebcf935e46e695eb9a9b37bb9ced668676f975613ee6bbdcbe18663f9-json.log
-rw-r----- 1 root root  760 Sep 27 09:30 /var/lib/docker/containers/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722/d5e5a18b7908aae547232c9b14a11bd0c08c5cab8ecf5f57c8ac0de362dff722-json.log
-rw-r----- 1 root root  41M Sep 27 07:22 /var/lib/docker/containers/eb0bcd559f11d03821fa176a9782cdb255b8ec856e310f943ed59dbaae84649e/eb0bcd559f11d03821fa176a9782cdb255b8ec856e310f943ed59dbaae84649e-json.log
-rw-r----- 1 root root 4.0G Sep 27 09:53 /var/lib/docker/containers/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1/f0421174fee0b3192b66258fb6a37cf336ff569c61eddf432529721e7e63bea1-json.log
-rw-r----- 1 root root 811M Sep 27 09:53 /var/lib/docker/containers/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491/f4785884c666706104df5f8140f30fdaf4df1c345bdfdff1887345e966ba8491-json.log
-rw-r----- 1 root root  23M Sep 27 07:10 /var/lib/docker/containers/fcb032a9cc740de91e11f11f1c2bee2ca442b0b40458ecdd99a0556d52ffa621/fcb032a9cc740de91e11f11f1c2bee2ca442b0b40458ecdd99a0556d52ffa621-json.log

root@fgt-tst-pl:~# du -sh /var/lib/docker/containers/
5.9G    /var/lib/docker/containers/
root@fgt-tst-pl:~# docker ps -a
CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                        PORTS                            NAMES
fcb032a9cc74   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    pha1
226bf4318049   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    pha2
b5e9b2d8972a   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs1
843d2c8bd417   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs-takeda
eb0bcd559f11   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs-zuellig
596d2c2565d0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   9 months ago    Exited (137) 25 minutes ago                                    whs2
868627bbf8a2   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-bayer
c999718ebcf9   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-sanofi
99e64cead7c6   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-gsk
2cb0d8f95140   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-takeda
989c4ec69a17   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-roche
738f9547fd41   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-msd
9455722e74ea   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-novo-nordisk
090b06f15dcd   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Exited (137) 25 minutes ago                                    mah-pfizer
d5e5a18b7908   traefik:v2.6                               "/entrypoint.sh --lo…"   10 months ago   Up 24 minutes                 80/tcp, 0.0.0.0:8080->8080/tcp   traefik
f0421174fee0   joaoluispdm/pharmaledger-imi-fgt:v0.11.4   "docker-entrypoint.s…"   10 months ago   Up 24 minutes (healthy)       3009/tcp, 8080-8081/tcp          fgt-workspace
f4785884c666   vegardit/traefik-logrotate:latest          "/sbin/tini -- /bin/…"   10 months ago   Up 24 minutes                                                  traefik-logrotate

Attempting to startup as described in https://github.com/PharmaLedger-IMI/fgt-workspace/issues/127#issuecomment-1349909320

The startup seems to work.

The excessive use of disk seems in the overlay2 folder.

root@fgt-tst-pl:~# du -sh /var/lib/docker/*
104K    /var/lib/docker/buildkit
5.9G    /var/lib/docker/containers
4.0K    /var/lib/docker/engine-id
58M /var/lib/docker/image
136K    /var/lib/docker/network
29G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
1.3G    /var/lib/docker/volumes

Like @TiagoV-PDMFC said, to diagnose this, the image filesystem should be put in read-only, and we should see what fails.

(Please note in the next comment that overlay2 grew a lot while starting the dockers - builds).

joaoluis-pdm commented 1 year ago

Get some extra space meanwhile

root@fgt-tst-pl:~# docker image prune --all
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.2
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:5df868fe2e3d953a72083cf69ff1c615beb8d67daa56228d3bad571d7b0403f3
deleted: sha256:bf6ddbe7622073b40fd32081fe54889e72736623b195f4405e4b4dcdd6ae0588
deleted: sha256:d7d380fb6caebc8cec7e1909edba6160c48f0a38126a98e4f13b568df877806d
deleted: sha256:f67dc1ac7b52418c9930cdf18fbabca387c798f4360e0ff42a87e36649ab2710
deleted: sha256:b6d25141c84f1291934d196071c055811dd9fdf7c15c2b26020f2067c16865bd
deleted: sha256:5007036c423e808739707dd54f159cbf9f5513e931ad2b551aef6258e95fa4ae
deleted: sha256:8fb23a23af86f09605f9888ad9b1cbe454c60d952223f69437d68c09fd4a3d19
deleted: sha256:14b4415383cec797eb5376cfa7382ef3430b5f504d05e7b56e950c6a986f1a01
deleted: sha256:975ab0f9f82959dd093fb9deab1bc8026150989960e2f1db79d85533d3bf6694
deleted: sha256:56538c10778a0a59a6ff6101893ffd3623b471a6f6be2a2acddc5b1b6a190f44
deleted: sha256:22a2a13a6d015587199a8b604f53e64df2f55e4933a6617ea719b401f3d688b7
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.3
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:31aa14318ff1de2f8cc8ae2cca424f8fee9340cb779f1f8089372a72e7c676e5
deleted: sha256:ac4eee7292783561554318841e64d57e72562e2f330474cfe6c7995ff651f0f8
deleted: sha256:088fe30622359ede939a64cf3352cabeb56f75a869b10979f67f3c5ced203154
deleted: sha256:5b55ebddd2092a05b6b221eef195890f8a0afa038ef344660f3c2835deab9dfd
deleted: sha256:2ac584c79756aa97e22c95498c026fd3fd0b704c1a0385ff45ee52de757c6599
deleted: sha256:aa6b7318cc6c5bebda447e1f343d77a13d8b5b1660e12bd92d47cd08f80cbb53
deleted: sha256:0c29be2b8dc1c063ca62bb3c45538c628a591e4bf4d02c9ce4c44ccc2dc55b41
deleted: sha256:ea03144da2b0832e3dd2f48a95474b9b6c4376410f8b66bab8bd129e007681bf
deleted: sha256:fc09cef5f5babbdf0306204fbbb51623395966d97abc5fce64cf1fcb1d119189
deleted: sha256:149e0bdf13f6b609a8d0bb21827114ac4e04e09992fc2d272cc74dfb8b65eee2
deleted: sha256:ccacc1eb77a64077019006e357ff1f8ccfe2c14731822fe61ab4ebca23735382

Total reclaimed space: 4.618GB

root@fgt-tst-pl:~# docker image prune --all
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.2
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:5df868fe2e3d953a72083cf69ff1c615beb8d67daa56228d3bad571d7b0403f3
deleted: sha256:bf6ddbe7622073b40fd32081fe54889e72736623b195f4405e4b4dcdd6ae0588
deleted: sha256:d7d380fb6caebc8cec7e1909edba6160c48f0a38126a98e4f13b568df877806d
deleted: sha256:f67dc1ac7b52418c9930cdf18fbabca387c798f4360e0ff42a87e36649ab2710
deleted: sha256:b6d25141c84f1291934d196071c055811dd9fdf7c15c2b26020f2067c16865bd
deleted: sha256:5007036c423e808739707dd54f159cbf9f5513e931ad2b551aef6258e95fa4ae
deleted: sha256:8fb23a23af86f09605f9888ad9b1cbe454c60d952223f69437d68c09fd4a3d19
deleted: sha256:14b4415383cec797eb5376cfa7382ef3430b5f504d05e7b56e950c6a986f1a01
deleted: sha256:975ab0f9f82959dd093fb9deab1bc8026150989960e2f1db79d85533d3bf6694
deleted: sha256:56538c10778a0a59a6ff6101893ffd3623b471a6f6be2a2acddc5b1b6a190f44
deleted: sha256:22a2a13a6d015587199a8b604f53e64df2f55e4933a6617ea719b401f3d688b7
untagged: joaoluispdm/pharmaledger-imi-fgt:v0.11.3
untagged: joaoluispdm/pharmaledger-imi-fgt@sha256:31aa14318ff1de2f8cc8ae2cca424f8fee9340cb779f1f8089372a72e7c676e5
deleted: sha256:ac4eee7292783561554318841e64d57e72562e2f330474cfe6c7995ff651f0f8
deleted: sha256:088fe30622359ede939a64cf3352cabeb56f75a869b10979f67f3c5ced203154
deleted: sha256:5b55ebddd2092a05b6b221eef195890f8a0afa038ef344660f3c2835deab9dfd
deleted: sha256:2ac584c79756aa97e22c95498c026fd3fd0b704c1a0385ff45ee52de757c6599
deleted: sha256:aa6b7318cc6c5bebda447e1f343d77a13d8b5b1660e12bd92d47cd08f80cbb53
deleted: sha256:0c29be2b8dc1c063ca62bb3c45538c628a591e4bf4d02c9ce4c44ccc2dc55b41
deleted: sha256:ea03144da2b0832e3dd2f48a95474b9b6c4376410f8b66bab8bd129e007681bf
deleted: sha256:fc09cef5f5babbdf0306204fbbb51623395966d97abc5fce64cf1fcb1d119189
deleted: sha256:149e0bdf13f6b609a8d0bb21827114ac4e04e09992fc2d272cc74dfb8b65eee2
deleted: sha256:ccacc1eb77a64077019006e357ff1f8ccfe2c14731822fe61ab4ebca23735382

Total reclaimed space: 4.618GB

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  117G   41G   72G  37% /

pharmaledger@fgt-tst-pl:~/fgt-workspace/docker/api$ docker images
REPOSITORY                         TAG       IMAGE ID       CREATED         SIZE
joaoluispdm/pharmaledger-imi-fgt   v0.11.4   191e1d1e2c33   10 months ago   2.42GB
vegardit/traefik-logrotate         latest    56c93b36b4af   10 months ago   55.1MB
traefik                            v2.6      22c6901de2be   16 months ago   102MB

Although overlay2 grew 5GB :-( it seems we get more space for a while...