[BE] Audit Prod Storage

karntrehan commented 2 months ago

We have used 5TB storage space on prod. We need to audit what service is hogging up so much memory and archive some data.

dhruv-1001 commented 2 months ago

@nikhil241 to delete migration backup files files.

karntrehan commented 2 months ago

Backup files deleted by @nikhil241. 5.3TB being used. @Ajay-Maury and @nikhil241 to connect today to analyse this further.

Break of what is using how much space -> App DB, Posthog, UCI, etc..

karntrehan commented 2 months ago

Current storage pattern:

Total size: 7.0TB

Occupied size: 5.5TB

Size left: 1.5TB

Time left: 3 - 5 months

335G NL-old
363G nl-tsdb-backups
577G posthog-data
110M posthog-deploy
286G timescaledb-new
272G timescaledb
114G uci-services
36G backups
273G nl-tsdb-17thjan-backup

docker-lib ->

329G containers
2.6T volumes (postgres data + logs)
598M image
283G overlay2

AIs:

[x] 1.5 TB logs of postgres (home/remote/docker-lib/volumes/hasura_tdb-data/_data/log) to be cleared. Log rotation of 10 days to be added - @nikhil241
[x] nl-tsdb-17thjan-backup to be deleted - @Ajay-Maury
[x] timescaledb-new and timescaledb to be checked and whichever is not used, to be deleted - @Ajay-Maury
[x] ~nl-tsdb-backups must be older backups when we did not have a separate system. These to be checked and deleted - @Ajay-Maury~
[x] Posthog event table cleanup - Archieve more than 6 months data. Approx saving: 400 GB - @Ajay-Maury
[x] Posthog logs rotation - @Ajay-Maury
[x] Audit UCI storage - @chinmoy12c. Approx saving: Unknown.
[x] Prune unused docker images @nikhil241 Total reclaimed space: 32.93GB
[x] nl-tsdb-backups should not be deleted because It is mounted to the docker backup. ~- [ ] Need to plan out a separate directory for all the docker-mounted folders.~

Post clean-up

Cleanup size: ~3.5TB

New occupied size: ~2.1TB

Free size: ~4.9TB

Time left: ~12 months.

dhruv-1001 commented 2 months ago

@dhruv-1001 @karntrehan @Ajay-Maury to connect on tsdb.

dhruv-1001 commented 2 months ago

@dhruv-1001 @Ajay-Maury to audit and remove duplicate tables. And list out learnings.

TheNPDev commented 2 months ago

4th point cannot be deleted

chinmoy12c commented 2 months ago

Disk Space Utilization Analysis

Total Folder Space - 97 GB
- Stale Backup (data-backup) - 46 GB
- Growth Rate - Stagnant
- Last Modified - Feb 15
- Can be deleted
- Stateful Data (Databases) - 52 GB
- Growth Rate - Not Exponential
- Last Modified - Live Modification (stores db data)
- Stateless Services - 115 MB
  - Growth Rate - Stagnant
  - Last Modified - Whenever new services need to be deployed

Docker root filesystem container folder space ( ~ 2TB)

1.3T  volumes
353G  containers
256G  overlay2
429M  image
123M  swarm
73M   buildkit
612K  network
16K   plugins
8.0K  tmp
4.0K  trust
4.0K  runtimes
4.0K  engine-id

Container Folder mostly contains Log files that takes up space (~300 GB)

214G  ./8a7c659f2c737b72ccba50612dbd8f4f923a08faeaa7e855c16d369625065fd7/8a7c659f2c737b72ccba50612dbd8f4f923a08faeaa7e855c16d369625065fd7-json.log
62G   ./0fa902bce298a6fb9067ba32821e56a59fd06670071775c891f18c825707d730/0fa902bce298a6fb9067ba32821e56a59fd06670071775c891f18c825707d730-json.log
23G   ./97b612cb5df2d299434ff87e58d1ba112afb1b1d4255abdc75b6974dec63797b/97b612cb5df2d299434ff87e58d1ba112afb1b1d4255abdc75b6974dec63797b-json.log
11G   ./204c9d67d7dc1e5af03549d85f2ae1ddd1a30b1160fa9cd8698ccd22b31562dc/204c9d67d7dc1e5af03549d85f2ae1ddd1a30b1160fa9cd8698ccd22b31562dc-json.log
9.9G  ./7c146b0f7cfceacc941ed5d95825aad2732f4f16747b3036ce34b13f416386a9/7c146b0f7cfceacc941ed5d95825aad2732f4f16747b3036ce34b13f416386a9-json.log
6.1G  ./daa2bc3548b4995d615b6e196b48eec325ba2b2418f18ac215cec965199acaae/daa2bc3548b4995d615b6e196b48eec325ba2b2418f18ac215cec965199acaae-json.log
5.4G  ./336624671d80485bb6b161867dd8a782dda76ea3daba99bd1b2f09e089291020/336624671d80485bb6b161867dd8a782dda76ea3daba99bd1b2f09e089291020-json.log
4.5G  ./4826081d8e058f2d09e5c0efff18b397da3ff3029164035b4cfdf3472e75ea70/4826081d8e058f2d09e5c0efff18b397da3ff3029164035b4cfdf3472e75ea70-json.log
3.3G  ./1261143b16720b1fe7d9fb9fabd1c727ef9470a5af510a6a62750429bd41f80c/1261143b16720b1fe7d9fb9fabd1c727ef9470a5af510a6a62750429bd41f80c-json.log
3.1G  ./28989efe8e681db4ddcf67011673b30d67daecc4e17d4253f9889cce81cef340/28989efe8e681db4ddcf67011673b30d67daecc4e17d4253f9889cce81cef340-json.log
1.7G  ./2c8d5ce822b28b81c379b8f1325ac7399329da6ac880b060d1c783791023da1e/2c8d5ce822b28b81c379b8f1325ac7399329da6ac880b060d1c783791023da1e-json.log
1.2G  ./62529cbdb63e614f42f7f73f8e13647f1db33d2ef01f8fdca0955f91668c1663/62529cbdb63e614f42f7f73f8e13647f1db33d2ef01f8fdca0955f91668c1663-json.log
1.1G  ./6da634efc34b61de3feb959def5352b6d348e51f964014a2aaffe6a74a59b6e1/6da634efc34b61de3feb959def5352b6d348e51f964014a2aaffe6a74a59b6e1-json.log
1.1G  ./5b06c0b6bf88a7775404de15f7ebca401c5ee484ada07aff7e6f4a65368af6af/5b06c0b6bf88a7775404de15f7ebca401c5ee484ada07aff7e6f4a65368af6af-json.log
1.1G  ./1ff1ae8d8d8f0870283c24ffb7382cf8f1e97fd76f942f750ebad514577919f8/1ff1ae8d8d8f0870283c24ffb7382cf8f1e97fd76f942f750ebad514577919f8-json.log
1.1G  ./184214b57a99b392b682198a16030136b7abb00d30e7644d5cb75976cf0a6f71/184214b57a99b392b682198a16030136b7abb00d30e7644d5cb75976cf0a6f71-json.log
1.1G  ./0d6dac896e71f50b31d28785955d8bbfd87fcc320bbaba3224445df1d1e48763/0d6dac896e71f50b31d28785955d8bbfd87fcc320bbaba3224445df1d1e48763-json.log
1.1G  ./0a19075a8cdfd284f483003a81ef1f43053a41d1744c249e93d83c23db00f870/0a19075a8cdfd284f483003a81ef1f43053a41d1744c249e93d83c23db00f870-json.log
534M  ./cf03883c9e2c9f79a74c25481fa819a01d02f370c5d2cf35bf9453d1d6978b97/cf03883c9e2c9f79a74c25481fa819a01d02f370c5d2cf35bf9453d1d6978b97-json.log
517M  ./91e74a8ae9a4397603d00b0fd6e55ebebe3fb526abeadcf071507114f75c982a/91e74a8ae9a4397603d00b0fd6e55ebebe3fb526abeadcf071507114f75c982a-json.log
404M  ./ed6e1fe24087004941d01529ec65e6723d6ebcf10c49fe1886b680123e6499fb/ed6e1fe24087004941d01529ec65e6723d6ebcf10c49fe1886b680123e6499fb-json.log
403M  ./cefe2734ff5e06b2838d0771f78c05a42cefdad6a4598d1723db45617545e3e5/cefe2734ff5e06b2838d0771f78c05a42cefdad6a4598d1723db45617545e3e5-json.log
292M  ./18a0e5e56959ab224595f05a038d8209de7389b53054ecc2361607088e304488/18a0e5e56959ab224595f05a038d8209de7389b53054ecc2361607088e304488-json.log
290M  ./c1e7e145511c3801f5deb916b91dc2a3a5a30fd83d89246050887c9a341ebfb4/c1e7e145511c3801f5deb916b91dc2a3a5a30fd83d89246050887c9a341ebfb4-json.log
258M  ./e933a402d18af6e7fdef3d33e6b19b7717514f21b91facfaabba9653af8e0bd7/e933a402d18af6e7fdef3d33e6b19b7717514f21b91facfaabba9653af8e0bd7-json.log
257M  ./9c4e8a030d62c4eb35af5b9ca643c4bd19b5cbff2f176bd68ea43050cfe3388e/9c4e8a030d62c4eb35af5b9ca643c4bd19b5cbff2f176bd68ea43050cfe3388e-json.log
256M  ./fc8eed2bc7bf1039c65a0b2e1ce479a4f57ef380ee293a612aecf95ca09acb34/fc8eed2bc7bf1039c65a0b2e1ce479a4f57ef380ee293a612aecf95ca09acb34-json.log
254M  ./5b374522ca24c630f8e4c33fd2b80f047a53b728220f26988cfd791469577bcb/5b374522ca24c630f8e4c33fd2b80f047a53b728220f26988cfd791469577bcb-json.log
254M  ./5970a4c13d3a8af97ed6944b8ff2efb43e8bfecde2a9dc4fdff03ed3681c463c/5970a4c13d3a8af97ed6944b8ff2efb43e8bfecde2a9dc4fdff03ed3681c463c-json.log
151M  ./223cd26a3c5321a4888da3e7e038f39643d7fe37beb3496366c3b7bdb4259e9e/223cd26a3c5321a4888da3e7e038f39643d7fe37beb3496366c3b7bdb4259e9e-json.log
148M  ./ad0fb75f3e99f1c16c50ba0a4941aeebf56864e1e915b8921ca5c2e62b8f6596/ad0fb75f3e99f1c16c50ba0a4941aeebf56864e1e915b8921ca5c2e62b8f6596-json.log
128M  ./687d6deb53344062d1c693cde29c6c08ae6eca425d85583816852ba3f7e6dc70/687d6deb53344062d1c693cde29c6c08ae6eca425d85583816852ba3f7e6dc70-json.log
68M   ./ea682e6b4605d38738663b18463aa7512204587dbf421f32c476960b747c99cf/ea682e6b4605d38738663b18463aa7512204587dbf421f32c476960b747c99cf-json.log
59M   ./d85410c05b8a3161dc3dfd3b60f08c91e58cf717e85197acf69b594134308a9b/d85410c05b8a3161dc3dfd3b60f08c91e58cf717e85197acf69b594134308a9b-json.log
59M   ./779a366b94a0879a16c29c4ab5db4f722b9023962a8fc95322ca33c6a27a3197/779a366b94a0879a16c29c4ab5db4f722b9023962a8fc95322ca33c6a27a3197-json.log
59M   ./45784a9fa813087a0d8b8392fffac4587d39f5d3b357f55dcdff7ac83fe15764/45784a9fa813087a0d8b8392fffac4587d39f5d3b357f55dcdff7ac83fe15764-json.log
59M   ./0a73cb0f2d5a72200555aeafc769ab637b7edd569327203fdaab7fb26a0051ed/0a73cb0f2d5a72200555aeafc769ab637b7edd569327203fdaab7fb26a0051ed-json.log
48M   ./804eec3182502f4782eb2174a7903fb5b1f235badabb1426ffc5bc2c721650e8/804eec3182502f4782eb2174a7903fb5b1f235badabb1426ffc5bc2c721650e8-json.log
33M   ./3b2ee36a074d943ffa8dbea26dfe7b2980f737d0585f05035f53e27629249929/3b2ee36a074d943ffa8dbea26dfe7b2980f737d0585f05035f53e27629249929-json.log
32M   ./a6f3a3985adde972a53b506f48a8b4ca3f9918bdcb84278a2a7361d88e9a3578/a6f3a3985adde972a53b506f48a8b4ca3f9918bdcb84278a2a7361d88e9a3578-json.log
24M   ./dfecf08ed9b5c076f1af3bff252344f38d7f69e2bb5f6db3a886b2dc0096905e/dfecf08ed9b5c076f1af3bff252344f38d7f69e2bb5f6db3a886b2dc0096905e-json.log
17M   ./e15a93bfb9df500e3e98d2c7209f210ff3e3801cdfa6b9a73f61f92852f3d869/e15a93bfb9df500e3e98d2c7209f210ff3e3801cdfa6b9a73f61f92852f3d869-json.log
14M   ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log
8.3M  ./29550f99af661f4a22ca5668aecdcd04a9d88c8810b22914ba2257d0602d7d56/29550f99af661f4a22ca5668aecdcd04a9d88c8810b22914ba2257d0602d7d56-json.log
7.6M  ./6ca157c83efcd5aeda9e907345c40448bd4d29dd3be72871e251e9e55d4855e5/6ca157c83efcd5aeda9e907345c40448bd4d29dd3be72871e251e9e55d4855e5-json.log
5.4M  ./11f2920533501eb38684b5d0c481bee18981f73aea9e62cf12930e0fda9bcf4d/11f2920533501eb38684b5d0c481bee18981f73aea9e62cf12930e0fda9bcf4d-json.log
4.2M  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs/container.log
3.1M  ./1dab8833ad3a41913b2a4d97327af3b22acf9d57100ea90661a9385af0091463/1dab8833ad3a41913b2a4d97327af3b22acf9d57100ea90661a9385af0091463-json.log
2.7M  ./2443664a086f2cc19ed76c3dc6f1e2e2c77c1bb99b902d754e12019426dbe46f/2443664a086f2cc19ed76c3dc6f1e2e2c77c1bb99b902d754e12019426dbe46f-json.log
2.6M  ./9b8e0a20f56cc8ebfb9f8c1ac1afd70121433b97bc57e215f48cb5ebb04d760b/9b8e0a20f56cc8ebfb9f8c1ac1afd70121433b97bc57e215f48cb5ebb04d760b-json.log
2.6M  ./6ead42bc703ba57184ea6176f564c9f5c59f5efbf9e8366d42585b379f0d38dd/6ead42bc703ba57184ea6176f564c9f5c59f5efbf9e8366d42585b379f0d38dd-json.log
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.8.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.7.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.6.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.5.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.4.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.3.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.29.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.28.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.24.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.23.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.21.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.20.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.2.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.19.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.18.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.17.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.16.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.15.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.12.gz
1.9M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.11.gz
1.9M  ./52baa8a2fac7f79916c851af5f845e23932effce86b9d10fa9cd24753b07f285/52baa8a2fac7f79916c851af5f845e23932effce86b9d10fa9cd24753b07f285-json.log
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.9.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.27.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.26.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.25.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.22.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.14.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.13.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.10.gz
1.8M  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs/container.log.1.gz
1.3M  ./33c3511cbb6e0fdf97f560e867fa1fadcd3f4c02808c083d04342d514804d0ed/33c3511cbb6e0fdf97f560e867fa1fadcd3f4c02808c083d04342d514804d0ed-json.log
1.1M  ./1351db6a6cfa36ba907bcf6ad4c333bb72dc5ae181cbf5c83e666a225e214ba0/1351db6a6cfa36ba907bcf6ad4c333bb72dc5ae181cbf5c83e666a225e214ba0-json.log
1020K ./2b221df2a0d50f4c1012cb73871b76169dde96ee30768e3e5a0bdfa89e7910da/2b221df2a0d50f4c1012cb73871b76169dde96ee30768e3e5a0bdfa89e7910da-json.log
976K  ./91c076c1801bb548c28cea74bd6ff65b16e2d800c9af4ad4de85459efa4525f7/91c076c1801bb548c28cea74bd6ff65b16e2d800c9af4ad4de85459efa4525f7-json.log
944K  ./76b0fb04ffd0053168bf40a053992c25662613a8087a9061c456359a6dbaca13/76b0fb04ffd0053168bf40a053992c25662613a8087a9061c456359a6dbaca13-json.log
868K  ./3a732cf0e6188cd5abc6617a333f50575b79346c05f6729825ae463b54d8e492/3a732cf0e6188cd5abc6617a333f50575b79346c05f6729825ae463b54d8e492-json.log
728K  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs/container.log.4.gz
704K  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs/container.log.3.gz
684K  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs/container.log.2.gz
668K  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs/container.log.1.gz
620K  ./e386fa48fc52f3af1d6e7399a6c75df97e6a0a1e3db4a641e90d7ccc280de8fd/e386fa48fc52f3af1d6e7399a6c75df97e6a0a1e3db4a641e90d7ccc280de8fd-json.log
544K  ./e5f652e8d0a83921486bb21e314f6b2697fd6313787279601a6a06a69b2ea961/e5f652e8d0a83921486bb21e314f6b2697fd6313787279601a6a06a69b2ea961-json.log
500K  ./e40a3c6c28c4340ae6cc0b49cc231e7f53ffdf4d335d38fdf05f9d3c3167ce3a/e40a3c6c28c4340ae6cc0b49cc231e7f53ffdf4d335d38fdf05f9d3c3167ce3a-json.log
276K  ./bba981ca2c57e69110e64479eca930c522d6fb56ee705eee664d5eb6c0bd95a2/bba981ca2c57e69110e64479eca930c522d6fb56ee705eee664d5eb6c0bd95a2-json.log
272K  ./7ace1a04c9df6bc8b23e51c709c6cd9b1061780af831be1ab0280276a50e6ef3/7ace1a04c9df6bc8b23e51c709c6cd9b1061780af831be1ab0280276a50e6ef3-json.log
268K  ./8f696e051a0174d43705577a43a713944eb7f7f4377e99671baa0b16943e27be/8f696e051a0174d43705577a43a713944eb7f7f4377e99671baa0b16943e27be-json.log
100K  ./3fbe63942493fc9c05f1b97cb03872e0ad02f6ce04a5556fcbb3146d204b41e9/3fbe63942493fc9c05f1b97cb03872e0ad02f6ce04a5556fcbb3146d204b41e9-json.log
92K   ./bddd66a5db259d3620760c3230598b7bbcf612366d73cf67ae9c2414a8535d4d/bddd66a5db259d3620760c3230598b7bbcf612366d73cf67ae9c2414a8535d4d-json.log
84K   ./e2f856fbc44cd5269e0dcfa4f7bb03d393a5323209c6e2f70fb123e86ad0cd28/e2f856fbc44cd5269e0dcfa4f7bb03d393a5323209c6e2f70fb123e86ad0cd28-json.log
60K   ./dac12211a98793dc7b29a1d7800eaf177e8ae230e5544d0bedc50121a77ae14d/dac12211a98793dc7b29a1d7800eaf177e8ae230e5544d0bedc50121a77ae14d-json.log
28K   ./a720bec37d7a349a9f66f242e6edb8930e1755b9c8507d5e8427830fdc0122e8/a720bec37d7a349a9f66f242e6edb8930e1755b9c8507d5e8427830fdc0122e8-json.log
28K   ./a3815ca6776d58928b372f25f5e00952df5922d5ee74e413f13d597af1decd2b/a3815ca6776d58928b372f25f5e00952df5922d5ee74e413f13d597af1decd2b-json.log
16K   ./b0399a07ec3da51bfb0819562c55967cb2a66fa378e3863e52ebb79e3605a758/b0399a07ec3da51bfb0819562c55967cb2a66fa378e3863e52ebb79e3605a758-json.log
16K   ./40752761b1dc78d6267381746a4c3290d8562c8b086d64b9b185960dc108a822/40752761b1dc78d6267381746a4c3290d8562c8b086d64b9b185960dc108a822-json.log
12K   ./f88afdbe14ff39ebfbb25a6d511796e3abe1b67931dc9cb42470f061b3011227/f88afdbe14ff39ebfbb25a6d511796e3abe1b67931dc9cb42470f061b3011227-json.log
12K   ./b6cf889895f2f8f51df1ad99934b68f25b81fc734f258e5869b88862ae7cfd49/b6cf889895f2f8f51df1ad99934b68f25b81fc734f258e5869b88862ae7cfd49-json.log
12K   ./7dd463b9f42427fae15ba4a5c18398d339cd495512819dae48e13c73482ec5af/7dd463b9f42427fae15ba4a5c18398d339cd495512819dae48e13c73482ec5af-json.log
12K   ./6914397dd588c5a6130bf2647c2f70aaca1e99b56e8f773464f8788e0027f2bb/6914397dd588c5a6130bf2647c2f70aaca1e99b56e8f773464f8788e0027f2bb-json.log
12K   ./256256f09b267e3650ea0da666e25e1b7e07913a98ece6f13c241c0f2e652ff9/256256f09b267e3650ea0da666e25e1b7e07913a98ece6f13c241c0f2e652ff9-json.log
8.0K  ./d4835227e3334f5293b4af92889a77c213a191fc993063d1af048586f9bab855/d4835227e3334f5293b4af92889a77c213a191fc993063d1af048586f9bab855-json.log
8.0K  ./d3afd36e616c75071b19bed7eca54fbe3774ee01b328cba5281693b8352f4379/d3afd36e616c75071b19bed7eca54fbe3774ee01b328cba5281693b8352f4379-json.log
8.0K  ./bd76d86ef631e437bb3583e92f5522c2321210fec490b4ab0e788dff2bfd66df/bd76d86ef631e437bb3583e92f5522c2321210fec490b4ab0e788dff2bfd66df-json.log
4.0K  ./ca600755bd5bacbecd3962e066ac6e6a8a0895fd7f7c9890f8bd4e96e01ad5ec/ca600755bd5bacbecd3962e066ac6e6a8a0895fd7f7c9890f8bd4e96e01ad5ec-json.log
4.0K  ./8f15077eff89d0ddbe48605a395698b6f486a26291fb4cc177278d068b2bddde/local-logs
4.0K  ./678fe3063e5559fdcfeb6e4f36e7361c8c82a0ee551e055fb81bd62b368bdd29/678fe3063e5559fdcfeb6e4f36e7361c8c82a0ee551e055fb81bd62b368bdd29-json.log
4.0K  ./64f166a2a261f604fda10d53320c3a9cc4bb932bf512b4bf242b30f382df41c2/local-logs
4.0K  ./3ee7438deee2a1ee118cc901bcc8d6542f3d023c254c6e8042f1e3234b6c482a/3ee7438deee2a1ee118cc901bcc8d6542f3d023c254c6e8042f1e3234b6c482a-json.log
4.0K  ./2e7eda2a692253638264347b9e8e419458dbd6f407b9155766997d1017175e4f/2e7eda2a692253638264347b9e8e419458dbd6f407b9155766997d1017175e4f-json.log
4.0K  ./2a4c51005881f159d47903574bfd9379958d74af13af5b7d0b32b9e5a8a950ad/2a4c51005881f159d47903574bfd9379958d74af13af5b7d0b32b9e5a8a950ad-json.log
4.0K  ./21f1f420848acb218bc4f5c1f09f174df849131c9757f833b6226fdbcf120dde/21f1f420848acb218bc4f5c1f09f174df849131c9757f833b6226fdbcf120dde-json.log
4.0K  ./1315b58112ae5546454cccfd8154234cd53df4236bc6fbe927a7c21392e83eee/1315b58112ae5546454cccfd8154234cd53df4236bc6fbe927a7c21392e83eee-json.log
0 ./1ce717df9cb1b39f04abde8035d75fbe238b15dc73a430e919306a33b606176a/1ce717df9cb1b39f04abde8035d75fbe238b15dc73a430e919306a33b606176a-json.log

Services taking up most space:
- Outbound (214G)
- Orchestrator (62GB)
- Inbound (23GB)
- Broadcast Transformer (11GB)
- Transformer (9.9GB)
- Posthog (6.1GB)
- Templater (5.4GB)
- Aggregate (4.5G)

Final Results

data-backup to be removed (One-Time Manual) (46 GB)
In-Container Logs to be removed (One-Time Manual, this should not grow now since this has been fixed by implementing log rotation in services).

Docker Logs to be removed ( This has to be done through cron ).


214G   ./8a7c659f2c737b72ccba50612dbd8f4f923a08faeaa7e855c16d369625065fd7/8a7c659f2c737b72ccba50612dbd8f4f923a08faeaa7e855c16d369625065fd7-json.log
62G   ./0fa902bce298a6fb9067ba32821e56a59fd06670071775c891f18c825707d730/0fa902bce298a6fb9067ba32821e56a59fd06670071775c891f18c825707d730-json.log
23G   ./97b612cb5df2d299434ff87e58d1ba112afb1b1d4255abdc75b6974dec63797b/97b612cb5df2d299434ff87e58d1ba112afb1b1d4255abdc75b6974dec63797b-json.log
11G   ./204c9d67d7dc1e5af03549d85f2ae1ddd1a30b1160fa9cd8698ccd22b31562dc/204c9d67d7dc1e5af03549d85f2ae1ddd1a30b1160fa9cd8698ccd22b31562dc-json.log
9.9G  ./7c146b0f7cfceacc941ed5d95825aad2732f4f16747b3036ce34b13f416386a9/7c146b0f7cfceacc941ed5d95825aad2732f4f16747b3036ce34b13f416386a9-json.log
6.1G  ./daa2bc3548b4995d615b6e196b48eec325ba2b2418f18ac215cec965199acaae/daa2bc3548b4995d615b6e196b48eec325ba2b2418f18ac215cec965199acaae-json.log
5.4G  ./336624671d80485bb6b161867dd8a782dda76ea3daba99bd1b2f09e089291020/336624671d80485bb6b161867dd8a782dda76ea3daba99bd1b2f09e089291020-json.log
4.5G  ./4826081d8e058f2d09e5c0efff18b397da3ff3029164035b4cfdf3472e75ea70/4826081d8e058f2d09e5c0efff18b397da3ff3029164035b4cfdf3472e75ea70-json.log```

karntrehan commented 2 months ago

WIP. To be closed by 3rd.

nikhil241 commented 2 months ago

To clean up logs in PostgreSQL, we need to write a script and schedule it using a cron job.

dhruv-1001 commented 2 months ago

Actions items.

[ ] Mounting docker volumes to specific folder. Add user based access to directory.
[ ] To clean up logs in PostgreSQL, through script.

dhruv-1001 commented 2 months ago

@chinmoy12c to list out what all can be deleted. @chinmoy12c @nikhil241 to connect on cron. Need to delete NL & UCI logs.

nikhil241 commented 1 month ago

A cron job has been set up for Postgres logs for the time being.

dhruv-1001 commented 1 month ago

Current storage pattern:

Total size: 7.0TB

Occupied size: 2.9TB

Size left: 3.8TB

Time left: 9 - 15 months

karntrehan commented 1 month ago

UCI cleanups to be added to cron.

Can be closed post addition.

Mission-Prerna / nl-apis