search

scientist-softserv / adventist_knapsack

Apache License 2.0
1 stars 0 forks source link

🐛 Adventist out of disk space #199

Closed ShanaLMoore closed 1 month ago

ShanaLMoore commented 4 months ago

Summary

We had to increase the pvc volume for Adventist because it ran out of disk space. We need to find the root cause of why it's filling up though.

Creating this ticket on Rob's request.

jeremyf commented 4 months ago

Well, this the first time I've been assigned or worked on this kind of ticket.

ShanaLMoore commented 4 months ago

Well, this the first time I've been assigned or worked on this kind of ticket.

@jeremyf my bad! that was meant for rob 🙈

orangewolf commented 4 months ago

this issue was /home/postgres/pgdata/pgroot/pg_log getting full. according to https://github.com/zalando/postgres-operator/issues/431 this is because of excess logging. Likely what is happening is that we are just writing a ton of data each day. we know that's the case because we are trying to import records as fast as we can. We do need to get alerts when the disk starts to fill up, but otherwise there isn't much to fix here.