search

Seneca-CDOT / telescope

A tool for tracking blogs in orbit around Seneca's open source involvement
https://telescope.cdot.systems
BSD 2-Clause "Simplified" License
94 stars 188 forks source link

Figure out which container is consuming all available disk space #3571

Open humphd opened 2 years ago

humphd commented 2 years ago

From @manekenpix on Slack:

so, something is eating up our disk space on staging, but I'm not sure what it is, but I know that if I stop telescope and relaunch it, we get a big chunk of disk space back. I don't think it's related to dangling images or volumes since we haven't merged anything in a while. Telescope had been running for 36 hours straight, and I had to shut it down and back up to get some disk space back. I'll file an issue when I have more info from what I can see, it's a container that stores something in itself (probably not using a volume) and it gets deleted when the container is destroyed (shut down telescope). This also happens on production, but we have a lot more free disk space on production, so it's not an issue (on staging, we get to 95% disk usage in less than 2 days)

I did a quick scan through the containers on Portainer, and I can't figure it out. For example, going to https://dev.portainer.telescope.cdot.systems/#!/1/docker/containers/b01f331167b374581f7c56b2cf7ae17a47d8887d6bbbbda6c891fb803ff98687/stats shows the CPU, Memory, Network and I/O usage for the rss-bridge service. Each container can show you the same.

None of the ones I looked at showed any significant disk I/O.

humphd commented 2 years ago

Running everything locally, I see:

docker ps --size
CONTAINER ID   IMAGE                                                  COMMAND                   CREATED          STATUS                    PORTS                                        NAMES                  SIZE
63133c7f496e   telescope_api_parser                                   "docker-entrypoint.s…"    25 minutes ago   Up 24 minutes (healthy)                                                parser                 0B (virtual 154MB)
cdb734cfdaec   telescope_api_nginx                                    "/docker-entrypoint.…"    25 minutes ago   Up 25 minutes             80/tcp, 0.0.0.0:8000->8000/tcp               nginx                  1.09kB (virtual 47.4MB)
87993a030850   telescope_api_sso                                      "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                sso                    0B (virtual 131MB)
3663d604b16a   telescope_api_posts                                    "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                posts                  0B (virtual 143MB)
d64175c8a819   telescope_api_status                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)   0.0.0.0:1111->1111/tcp                       status                 0B (virtual 214MB)
1d6a562b5bbe   telescope_api_image                                    "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                image                  25.2MB (virtual 182MB)
30c5a6bb600c   telescope_api_search                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                search                 0B (virtual 127MB)
682406d53e3e   telescope_api_planet                                   "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)   0.0.0.0:9876->9876/tcp                       planet                 0B (virtual 135MB)
07b739881b5d   telescope_api_dependency-discovery                     "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                dependency-discovery   0B (virtual 136MB)
ecea645da632   telescope_api_feed-discovery                           "docker-entrypoint.s…"    25 minutes ago   Up 25 minutes (healthy)                                                feed-discovery         0B (virtual 129MB)
86f7d8f713b9   nginx:stable-alpine                                    "/docker-entrypoint.…"    10 days ago      Up 25 minutes             0.0.0.0:8888->80/tcp                         test-web-content       1.09kB (virtual 21.8MB)
df9193312ef2   traefik:v2.6.1                                         "/entrypoint.sh --lo…"    10 days ago      Up 25 minutes             0.0.0.0:80->80/tcp, 0.0.0.0:8080->8080/tcp   traefik                0B (virtual 99.6MB)
d8b23a9a30ee   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   12 days ago      Up About an hour          5000/tcp                                     supabase-storage       36kB (virtual 139MB)
76bfc097adb1   supabase/gotrue:v2.5.22                                "gotrue"                  12 days ago      Up About an hour                                                       supabase-auth          0B (virtual 32MB)
561844e31429   postgrest/postgrest:v9.0.0                             "/bin/sh -c postgrest"    12 days ago      Up About an hour          3000/tcp                                     supabase-rest          0B (virtual 353MB)
371451d33a82   telescope_api_pg_backup                                "crond -f -l 8"           12 days ago      Up 25 minutes                                                          pg-backup-cron-job     2B (virtual 9.26MB)
9dc9d7579639   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    12 days ago      Up About an hour          8080/tcp                                     supabase-meta          152B (virtual 342MB)
63196f510c7b   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    12 days ago      Up About an hour                                                       supabase-realtime      3.9MB (virtual 190MB)
d2ca91c17ab3   kong:2.1                                               "/docker-entrypoint.…"    12 days ago      Up About an hour          8000-8001/tcp, 8443-8444/tcp                 supabase-kong          22.4kB (virtual 158MB)
ceb9f3897465   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    12 days ago      Up 25 minutes             80/tcp                                       rss-bridge             2.42kB (virtual 440MB)
501903f8ad7c   redis:latest                                           "docker-entrypoint.s…"    12 days ago      Up 25 minutes             0.0.0.0:6379->6379/tcp                       redis                  0B (virtual 107MB)
3b3a8c378b78   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    12 days ago      Up 25 minutes             0.0.0.0:9200->9200/tcp, 9300/tcp             elasticsearch          437MB (virtual 1.04GB)
c8b7ce584821   kristophjunge/test-saml-idp                            "docker-php-entrypoi…"    12 days ago      Up 25 minutes             80/tcp, 8443/tcp, 0.0.0.0:8081->8080/tcp     login                  96.3kB (virtual 457MB)
37f822b887a5   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    12 days ago      Up About an hour          0.0.0.0:5432->5432/tcp                       supabase-db            2.31GB (virtual 4.13GB)
5cc3b714d8d8   supabase/studio:latest                                 "docker-entrypoint.s…"    12 days ago      Up About an hour          0.0.0.0:8910->3000/tcp                       supabase-studio        152B (virtual 936MB)

If you scroll that all the way to the right, you can see the total size of the containers. Notice:

Everything else is tiny. I'll let this run for a while and see what it does. It would be interesting to see this same data for staging/prod.

manekenpix commented 2 years ago

staging:

CONTAINER ID   IMAGE                                                  COMMAND                   CREATED        STATUS                  PORTS                                                                      NAMES                  SIZE
835f0ae0b395   portainer/portainer-ce:alpine                          "/portainer -H unix:…"    23 hours ago   Up 23 hours             8000/tcp, 9000/tcp, 9443/tcp                                               portainer              0B (virtual 285MB)
2978bb78df56   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   23 hours ago   Up 23 hours             5000/tcp                                                                   supabase-storage       36kB (virtual 139MB)
274f0f9c318e   docker.cdot.systems/feed-discovery:staging             "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              feed-discovery         0B (virtual 131MB)
e856b8a2ac21   docker.cdot.systems/search:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              search                 0B (virtual 128MB)
7d29f2e1235c   docker.cdot.systems/parser:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              parser                 0B (virtual 155MB)
d13da562d41d   docker.cdot.systems/nginx:staging                      "/docker-entrypoint.…"    23 hours ago   Up 23 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx                  9.65kB (virtual 48.7MB)
63c816f9d792   docker.cdot.systems/posts:staging                      "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              posts                  0B (virtual 145MB)
7a9274cd87a2   docker.cdot.systems/status:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              status                 0B (virtual 216MB)
ee36fd13e68f   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    23 hours ago   Up 23 hours             8080/tcp                                                                   supabase-meta          55B (virtual 341MB)
1f393b54698b   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    23 hours ago   Up 23 hours                                                                                        supabase-realtime      3.85MB (virtual 199MB)
2f71bfb5b0a0   supabase/gotrue:v2.5.22                                "gotrue"                  23 hours ago   Up 23 hours                                                                                        supabase-auth          0B (virtual 33MB)
636d3022bdbc   green_pg_backup                                        "crond -f -l 8"           23 hours ago   Up 23 hours                                                                                        pg-backup-cron-job     2B (virtual 9.56MB)
fd58334ed328   postgrest/postgrest:v9.0.0                             "/bin/postgrest"          23 hours ago   Up 23 hours             3000/tcp                                                                   supabase-rest          0B (virtual 16.3MB)
1b112bac4b48   docker.cdot.systems/sso:staging                        "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              sso                    0B (virtual 133MB)
0548e462fcf9   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    23 hours ago   Up 23 hours             80/tcp                                                                     rss-bridge             2.42kB (virtual 475MB)
c20eceeeb782   kong:2.1                                               "/docker-entrypoint.…"    23 hours ago   Up 23 hours             8000-8001/tcp, 8443-8444/tcp                                               supabase-kong          22.4kB (virtual 147MB)
44ed2ede6c99   docker.cdot.systems/dependency-discovery:staging       "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              dependency-discovery   0B (virtual 137MB)
d6a13b2bbca1   docker.cdot.systems/image:staging                      "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              image                  25.2MB (virtual 178MB)
9683f1f9b796   bitnami/oauth2-proxy:7.2.1                             "oauth2-proxy --prov…"    23 hours ago   Up 23 hours             4180/tcp, 8080/tcp                                                         oauth2-proxy           0B (virtual 118MB)
ee7fee2df3d7   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    23 hours ago   Up 23 hours             5432/tcp                                                                   supabase-db            1.5GB (virtual 3.32GB)
c0c38eb749aa   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    23 hours ago   Up 23 hours             9200/tcp, 9300/tcp                                                         elasticsearch          80.9MB (virtual 692MB)
2e7286d4ee9a   traefik:v2.6.1                                         "/entrypoint.sh --ap…"    23 hours ago   Up 23 hours             80/tcp                                                                     traefik                0B (virtual 102MB)
116b07847a54   docker.cdot.systems/planet:staging                     "docker-entrypoint.s…"    23 hours ago   Up 23 hours (healthy)                                                                              planet                 0B (virtual 136MB)
4306d3d8189f   redis:latest                                           "docker-entrypoint.s…"    23 hours ago   Up 23 hours             6379/tcp                                                                   redis                  0B (virtual 117MB)
32ccde5c629a   certbot/certbot                                        "/bin/sh -c 'trap ex…"    23 hours ago   Up 23 hours             80/tcp, 443/tcp                                                            certbot                873kB (virtual 116MB)
ec6fe76dc7bb   supabase/studio:latest                                 "docker-entrypoint.s…"    23 hours ago   Up 23 hours             3000/tcp                                                                   supabase-studio        152B (virtual 968MB)

production:

$ docker ps --size
CONTAINER ID   IMAGE                                                  COMMAND                   CREATED        STATUS                  PORTS                                                                      NAMES                  SIZE
168533abae85   portainer/portainer-ce:alpine                          "/portainer -H unix:…"    39 hours ago   Up 39 hours             8000/tcp, 9000/tcp, 9443/tcp                                               portainer              0B (virtual 285MB)
e5d7f75f6ddc   docker.cdot.systems/feed-discovery:production          "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              feed-discovery         0B (virtual 130MB)
da61b3fe2934   supabase/storage-api:v0.10.0                           "/bin/sh -c \"./stora…"   39 hours ago   Up 39 hours             5000/tcp                                                                   supabase-storage       36kB (virtual 139MB)
f2675a752872   docker.cdot.systems/nginx:production                   "/docker-entrypoint.…"    39 hours ago   Up 39 hours             0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   nginx                  9.62kB (virtual 48.7MB)
8b6c1ede77df   docker.cdot.systems/parser:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              parser                 0B (virtual 154MB)
22a86eb8c100   docker.cdot.systems/search:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              search                 0B (virtual 127MB)
18c0dd0d9710   supabase/realtime:v0.19.3                              "bash -c './prod/rel…"    39 hours ago   Up 39 hours                                                                                        supabase-realtime      3.85MB (virtual 199MB)
e0445af4c9d4   supabase/gotrue:v2.5.22                                "gotrue"                  39 hours ago   Up 39 hours                                                                                        supabase-auth          0B (virtual 33MB)
ed7e8012bc35   supabase/postgres-meta:v0.33.3                         "docker-entrypoint.s…"    39 hours ago   Up 39 hours             8080/tcp                                                                   supabase-meta          55B (virtual 341MB)
6edc789d5736   green_pg_backup                                        "crond -f -l 8"           39 hours ago   Up 39 hours                                                                                        pg-backup-cron-job     2B (virtual 9.56MB)
aa22bfd7dcc9   postgrest/postgrest:v9.0.0                             "/bin/postgrest"          39 hours ago   Up 39 hours             3000/tcp                                                                   supabase-rest          0B (virtual 16.3MB)
a4e19313cd50   bitnami/oauth2-proxy:7.2.1                             "oauth2-proxy --prov…"    39 hours ago   Up 39 hours             4180/tcp, 8080/tcp                                                         oauth2-proxy           0B (virtual 101MB)
99679514b6d7   docker.cdot.systems/status:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              status                 0B (virtual 215MB)
82859090a87e   docker.cdot.systems/sso:production                     "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              sso                    0B (virtual 132MB)
35ab6d242e77   docker.cdot.systems/posts:production                   "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              posts                  0B (virtual 144MB)
76da9f586a72   docker.cdot.systems/dependency-discovery:production    "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              dependency-discovery   0B (virtual 136MB)
fde02c1224da   kong:2.1                                               "/docker-entrypoint.…"    39 hours ago   Up 39 hours             8000-8001/tcp, 8443-8444/tcp                                               supabase-kong          22.5kB (virtual 147MB)
447c5e9f0a97   rssbridge/rss-bridge:2022-01-20                        "docker-php-entrypoi…"    39 hours ago   Up 39 hours             80/tcp                                                                     rss-bridge             2.42kB (virtual 475MB)
6e699c86d046   docker.cdot.systems/image:production                   "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              image                  25.2MB (virtual 177MB)
2000da302374   supabase/postgres:14.1.0                               "docker-entrypoint.s…"    39 hours ago   Up 39 hours             5432/tcp                                                                   supabase-db            3.68GB (virtual 5.5GB)
58e227de0814   traefik:v2.6.1                                         "/entrypoint.sh --ap…"    39 hours ago   Up 39 hours             80/tcp                                                                     traefik                0B (virtual 102MB)
1d32d079f676   redis:latest                                           "docker-entrypoint.s…"    39 hours ago   Up 39 hours             6379/tcp                                                                   redis                  0B (virtual 113MB)
8de8ca986608   supabase/studio:latest                                 "docker-entrypoint.s…"    39 hours ago   Up 39 hours             3000/tcp                                                                   supabase-studio        152B (virtual 968MB)
16c1d7c852a0   docker.cdot.systems/planet:production                  "docker-entrypoint.s…"    39 hours ago   Up 39 hours (healthy)                                                                              planet                 0B (virtual 135MB)
5d166d245979   docker.elastic.co/elasticsearch/elasticsearch:7.17.2   "/bin/tini -- /usr/l…"    39 hours ago   Up 39 hours             9200/tcp, 9300/tcp                                                         elasticsearch          83.4MB (virtual 695MB)
c48cafff5f6c   certbot/certbot                                        "/bin/sh -c 'trap ex…"    39 hours ago   Up 39 hours             80/tcp, 443/tcp                                                            certbot                968kB (virtual 116MB)