b4mad / racing

Community-driven SimRacing data collection and analysis
https://b4mad.racing
GNU General Public License v3.0
22 stars 13 forks source link

pgo storage fills up too quick / different psql operator? #245

Open durandom opened 1 year ago

durandom commented 1 year ago

because backups are eating up our storage: https://github.com/CrunchyData/postgres-operator/issues/2531

Consider migration to https://cloudnative-pg.io/

https://blog.palark.com/cloudnativepg-and-other-kubernetes-operators-for-postgresql/

durandom commented 1 year ago

either increasing the backup frequency might help as in https://github.com/CrunchyData/postgres-operator/issues/2531#issuecomment-922349084

or setting archive_mode to off as in https://github.com/CrunchyData/postgres-operator/issues/2531#issuecomment-1022070211

goern commented 1 year ago

/priority backlog

durandom commented 1 year ago

Extend the PVC for psql by +2Gb and wait until the DB recovers. Then trigger a one off backup either by:

oc annotate postgrescluster db --overwrite postgres-operator.crunchydata.com/pgbackrest-backup="$(date)"

or oc create job --from=cronjob/db-repo1-full db-backup-full

Long term we need either

goern commented 1 year ago

implemented some altering, see https://github.com/b4mad/racing/blob/main/manifests/env/phobos/postgresql/alerting.yaml

alertmanager will send to https://discord.com/channels/850841654327640074/1106466539273732136