search

MaRDI4NFDI / portal-compose

docker-composer repo for mardi
https://portal.mardi4nfdi.de
GNU General Public License v3.0
3 stars 1 forks source link

Investigate disk size requirements of MaRDI instanze #464

Closed physikerwelt closed 5 months ago

physikerwelt commented 5 months ago

Describe the issue docker system df -v ...


Local Volumes space usage:

VOLUME NAME                                                        LINKS     SIZE
a7f14914da435b8509da13f217e4c26eef11c4d7161b089d4edf562e4bd1c482   1         750.5MB
docker_formulasearch_harvests                                      2         79.8kB
docker_goaccess_db                                                 1         0B
docker_query-service-data                                          1         41.01GB
docker_traefik-letsencrypt                                         1         275.8kB
52ee5abbb4196c9bb980534d987f45583bbebc2125303581c5d23008a83efecb   1         3.958MB
6b13bc5e14fec500837827e872ae7175bae2516a974dcf3099f7ca4fea27164f   1         0B
docker_mediawiki-mongo-data                                        1         528.8MB
docker_provider-logs                                               1         53.53MB
395a34e8270178e8daa37226acba52aa3dbb10c24698423ec1fd16008d2117b2   0         0B
4dbf7a36a8ea126c0d90f328ab9de46cb5b4741f79a8713d7d6f1377edc2d340   0         10.57GB
c579b8b8019148d9f81a00bcbbe1418598e84d04bad3fc6a2708218f6bf5c877   0         144MB
111dd11caafae9b81e9a25e8d2d91d0e82340d5ae72ee69661211c9e433d9965   1         0B
docker_apache_logs                                                 1         33B
docker_grafana_data                                                1         2.341MB
docker_portainer-data                                              1         17.62MB
docker_quickstatements-data                                        2         498B
docker_uptime-kuma                                                 1         56.43MB
docker_matomo                                                      1         58.81MB
docker_prometheus_data                                             1         326.8MB
docker_shared_mardi_wikibase                                       6         2.755kB
f4d8820fba4e85f8759aff325cb1a176461e0c1779f24ff4d5a7b5d92f891b13   1         0B
75c9eac614ae08fcc171a8976423f80399c541f10cd76347ef2f9f6b2e304a39   1         0B
docker_backend-logs                                                1         102.3MB
docker_cassandra-data                                              2         159.9MB
docker_goaccess_report                                             2         1.112kB
docker_mediawiki-mysql-data                                        1         271.1GB
f84048e3a193d05e2b666c5d2058ff727efac700a4675539d44b235569d879ab   0         0B
docker_backup-logs                                                 1         537B
docker_es-data                                                     1         41.33MB
docker_shared_mediawiki_images                                     2         12.36MB
docker_shibboleth                                                  0         0B
docker_elasticsearch-data                                          1         20.42GB
docker_es-logs                                                     1         5.099MB
physikerwelt commented 5 months ago

The size consuption seems to be quite reasonable with the exception of

docker_mediawiki-mysql-data                                        1         271.1GB

I think disk sizes only grow and never shrink and expect this is due to a large number of jobs that were stored in mysql.

physikerwelt commented 5 months ago

this problem was fixed in https://github.com/MaRDI4NFDI/portal-compose/commit/ca4bc670e99d298e4dc52fcd8b5dd806c896c93b (the binary logs are still there, but no new ones will be created)

physikerwelt commented 5 months ago

Also purged the logs


PURGE BINARY LOGS BEFORE '2024-02-03';
physikerwelt commented 5 months ago 

root@4bdde1897b4d:/var/lib/mysql# ls -lah
total 163G
drwxr-xr-x 6 mysql mysql  20K Feb  4 12:54 .
drwxr-xr-x 1 root  root  4.0K Feb  2 00:56 ..
-rw-rw---- 1 mysql mysql 2.2M Feb  4 12:53 aria_log.00000001
-rw-rw---- 1 mysql mysql   52 Feb  4 12:53 aria_log_control
-rw-rw---- 1 mysql mysql    9 Feb  3 09:42 ddl_recovery.log
-rw-rw---- 1 mysql mysql 653K Feb  3 09:41 ib_buffer_pool
-rw-rw---- 1 mysql mysql  32M Feb  4 12:54 ib_logfile0
-rw-rw---- 1 mysql mysql  69G Feb  4 12:54 ibdata1
-rw-rw---- 1 mysql mysql  12M Feb  3 10:04 ibtmp1
-rw-rw---- 1 mysql mysql 1.1G Feb  3 00:12 mardb-bin.000650
-rw-rw---- 1 mysql mysql 1.1G Feb  3 00:47 mardb-bin.000651
-rw-rw---- 1 mysql mysql 1.1G Feb  3 00:58 mardb-bin.000652
-rw-rw---- 1 mysql mysql 1.1G Feb  3 01:12 mardb-bin.000653
-rw-rw---- 1 mysql mysql 1.1G Feb  3 01:41 mardb-bin.000654
-rw-rw---- 1 mysql mysql 1.1G Feb  3 02:14 mardb-bin.000655
-rw-rw---- 1 mysql mysql 1.1G Feb  3 02:49 mardb-bin.000656
-rw-rw---- 1 mysql mysql 1.1G Feb  3 03:21 mardb-bin.000657
-rw-rw---- 1 mysql mysql 1.1G Feb  3 03:39 mardb-bin.000658
-rw-rw---- 1 mysql mysql 1.1G Feb  3 03:56 mardb-bin.000659
-rw-rw---- 1 mysql mysql 1.1G Feb  3 04:11 mardb-bin.000660
-rw-rw---- 1 mysql mysql 1.1G Feb  3 04:35 mardb-bin.000661
-rw-rw---- 1 mysql mysql 1.1G Feb  3 05:06 mardb-bin.000662
-rw-rw---- 1 mysql mysql 1.1G Feb  3 05:43 mardb-bin.000663
-rw-rw---- 1 mysql mysql 1.1G Feb  3 06:08 mardb-bin.000664
-rw-rw---- 1 mysql mysql 1.1G Feb  3 06:40 mardb-bin.000665
-rw-rw---- 1 mysql mysql 1.1G Feb  3 07:13 mardb-bin.000666
-rw-rw---- 1 mysql mysql 1.1G Feb  3 07:51 mardb-bin.000667
-rw-rw---- 1 mysql mysql 1.1G Feb  3 08:08 mardb-bin.000668
-rw-rw---- 1 mysql mysql 1.1G Feb  3 08:50 mardb-bin.000669
-rw-rw---- 1 mysql mysql 1.1G Feb  3 09:21 mardb-bin.000670
-rw-rw---- 1 mysql mysql 1.1G Feb  3 09:31 mardb-bin.000671
-rw-rw---- 1 mysql mysql 824M Feb  3 09:41 mardb-bin.000672
-rw-rw---- 1 mysql mysql 1.1G Feb  3 09:53 mardb-bin.000673
-rw-rw---- 1 mysql mysql 1.1G Feb  3 10:03 mardb-bin.000674
-rw-rw---- 1 mysql mysql 1.1G Feb  3 10:14 mardb-bin.000675
-rw-rw---- 1 mysql mysql 1.1G Feb  3 10:27 mardb-bin.000676
-rw-rw---- 1 mysql mysql 1.1G Feb  3 12:34 mardb-bin.000677
-rw-rw---- 1 mysql mysql 1.1G Feb  3 12:56 mardb-bin.000678
-rw-rw---- 1 mysql mysql 1.1G Feb  3 13:20 mardb-bin.000679
-rw-rw---- 1 mysql mysql 1.1G Feb  3 13:32 mardb-bin.000680
-rw-rw---- 1 mysql mysql 1.1G Feb  3 13:43 mardb-bin.000681
-rw-rw---- 1 mysql mysql 1.1G Feb  3 13:55 mardb-bin.000682
-rw-rw---- 1 mysql mysql 1.1G Feb  3 15:06 mardb-bin.000683
-rw-rw---- 1 mysql mysql 1.1G Feb  3 15:38 mardb-bin.000684
-rw-rw---- 1 mysql mysql 1.1G Feb  3 16:12 mardb-bin.000685
-rw-rw---- 1 mysql mysql 1.1G Feb  3 16:32 mardb-bin.000686
-rw-rw---- 1 mysql mysql 1.1G Feb  3 16:57 mardb-bin.000687
-rw-rw---- 1 mysql mysql 1.1G Feb  3 17:17 mardb-bin.000688
-rw-rw---- 1 mysql mysql 1.1G Feb  3 17:44 mardb-bin.000689
-rw-rw---- 1 mysql mysql 1.1G Feb  3 18:17 mardb-bin.000690
-rw-rw---- 1 mysql mysql 1.1G Feb  3 18:29 mardb-bin.000691
-rw-rw---- 1 mysql mysql 1.1G Feb  3 18:39 mardb-bin.000692
-rw-rw---- 1 mysql mysql 1.1G Feb  3 18:52 mardb-bin.000693
-rw-rw---- 1 mysql mysql 1.1G Feb  3 19:05 mardb-bin.000694
-rw-rw---- 1 mysql mysql 1.1G Feb  3 19:15 mardb-bin.000695
-rw-rw---- 1 mysql mysql 1.1G Feb  3 19:22 mardb-bin.000696
-rw-rw---- 1 mysql mysql 1.1G Feb  3 19:31 mardb-bin.000697
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:00 mardb-bin.000698
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:08 mardb-bin.000699
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:12 mardb-bin.000700
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:32 mardb-bin.000701
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:37 mardb-bin.000702
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:42 mardb-bin.000703
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:46 mardb-bin.000704
-rw-rw---- 1 mysql mysql 1.1G Feb  3 20:51 mardb-bin.000705
-rw-rw---- 1 mysql mysql 1.1G Feb  3 21:02 mardb-bin.000706
-rw-rw---- 1 mysql mysql 1.1G Feb  3 21:09 mardb-bin.000707
-rw-rw---- 1 mysql mysql 1.1G Feb  3 21:21 mardb-bin.000708
-rw-rw---- 1 mysql mysql 1.1G Feb  3 21:35 mardb-bin.000709
-rw-rw---- 1 mysql mysql 1.1G Feb  3 21:50 mardb-bin.000710
-rw-rw---- 1 mysql mysql 1.1G Feb  3 22:05 mardb-bin.000711
-rw-rw---- 1 mysql mysql 1.1G Feb  3 22:20 mardb-bin.000712
-rw-rw---- 1 mysql mysql 1.1G Feb  3 22:36 mardb-bin.000713
-rw-rw---- 1 mysql mysql 1.1G Feb  3 22:51 mardb-bin.000714
-rw-rw---- 1 mysql mysql 1.1G Feb  3 23:07 mardb-bin.000715
-rw-rw---- 1 mysql mysql 1.1G Feb  3 23:23 mardb-bin.000716
-rw-rw---- 1 mysql mysql 1.1G Feb  3 23:39 mardb-bin.000717
-rw-rw---- 1 mysql mysql 1.1G Feb  3 23:54 mardb-bin.000718
-rw-rw---- 1 mysql mysql 1.1G Feb  4 00:09 mardb-bin.000719
-rw-rw---- 1 mysql mysql 1.1G Feb  4 00:26 mardb-bin.000720
-rw-rw---- 1 mysql mysql 1.1G Feb  4 00:41 mardb-bin.000721
-rw-rw---- 1 mysql mysql 1.1G Feb  4 00:57 mardb-bin.000722
-rw-rw---- 1 mysql mysql 1.1G Feb  4 01:13 mardb-bin.000723
-rw-rw---- 1 mysql mysql 1.1G Feb  4 01:28 mardb-bin.000724
-rw-rw---- 1 mysql mysql 1.1G Feb  4 01:42 mardb-bin.000725
-rw-rw---- 1 mysql mysql 1.1G Feb  4 01:56 mardb-bin.000726
-rw-rw---- 1 mysql mysql 1.1G Feb  4 02:07 mardb-bin.000727
-rw-rw---- 1 mysql mysql 1.1G Feb  4 02:16 mardb-bin.000728
-rw-rw---- 1 mysql mysql 1.1G Feb  4 04:01 mardb-bin.000729
-rw-rw---- 1 mysql mysql 1.1G Feb  4 05:13 mardb-bin.000730
-rw-rw---- 1 mysql mysql 1.1G Feb  4 06:13 mardb-bin.000731
-rw-rw---- 1 mysql mysql 1.1G Feb  4 09:20 mardb-bin.000732
-rw-rw---- 1 mysql mysql 1.1G Feb  4 09:43 mardb-bin.000733
-rw-rw---- 1 mysql mysql 1.1G Feb  4 09:58 mardb-bin.000734
-rw-rw---- 1 mysql mysql 1.1G Feb  4 10:15 mardb-bin.000735
-rw-rw---- 1 mysql mysql 1.1G Feb  4 10:29 mardb-bin.000736
-rw-rw---- 1 mysql mysql 1.1G Feb  4 10:41 mardb-bin.000737
-rw-rw---- 1 mysql mysql 1.1G Feb  4 10:56 mardb-bin.000738
-rw-rw---- 1 mysql mysql 1.1G Feb  4 11:11 mardb-bin.000739
-rw-rw---- 1 mysql mysql 1.1G Feb  4 11:31 mardb-bin.000740
-rw-rw---- 1 mysql mysql 1.1G Feb  4 11:55 mardb-bin.000741
-rw-rw---- 1 mysql mysql 1.1G Feb  4 12:19 mardb-bin.000742
-rw-rw---- 1 mysql mysql 1.1G Feb  4 12:43 mardb-bin.000743
-rw-rw---- 1 mysql mysql 484M Feb  4 12:54 mardb-bin.000744
-rw-rw---- 1 mysql mysql 1.8K Feb  4 12:54 mardb-bin.index
-rw-rw---- 1 mysql mysql  256 Feb  3 09:42 mardb-relay-bin.000020
-rw-rw---- 1 mysql mysql   25 Feb  3 09:42 mardb-relay-bin.index
-rw-rw---- 1 mysql mysql  153 Feb  3 09:42 master.info
-rw-rw---- 1 mysql mysql    0 Oct 28  2022 multi-master.info
drwx------ 2 mysql mysql  12K Nov 25 14:16 my_wiki
drwx------ 2 mysql mysql 4.0K Nov  3 16:33 mysql
-rw-r--r-- 1 mysql mysql   16 Nov  3 16:33 mysql_upgrade_info
drwx------ 2 mysql mysql 4.0K Nov  3 16:33 performance_schema
-rw-rw---- 1 mysql mysql   36 Feb  3 09:42 relay-log.info
drwx------ 2 mysql mysql  12K Oct  6 06:00 wiki_swmath
root@4bdde1897b4d:/var/lib/mysql# du . -sh
289G    .