search

bcgov / NRPTI

Natural Resources Public Transparency Initiative
Apache License 2.0
5 stars 15 forks source link

Fix mongodb backups failing/crashing #1256

Closed sggerard closed 2 months ago

sggerard commented 2 months ago

Describe the Bug Nightly backups of the production mongodb database are failing.

Expected Behaviour Nightly backups should complete successfully

Actual Behaviour

Image

Implications No backups of critical production data. Prevents new work/changes from being deployed to production.

sggerard commented 2 months ago

Fixed by increasing the PVC storage from 10GB to 15GB and increasing the memory for the verification pod to 3gb from 2gb. Found the underlying cause to be the size of the 'audit' collection which stores every search query ever made. Will create a ticket to address