search

maxisam / mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
https://maxisam.github.io/mgob/
MIT License
123 stars 16 forks source link

Feature request: always preserve 1 version of the log #104

Open aneagoe opened 3 months ago

aneagoe commented 3 months ago

After upgrading to 2.0.20, I noticed a lot of failed backups the next day, but unfortunately the pod log didn't say much:

time="2024-03-13T02:31:00Z" level=info msg="Backup started" plan=kviz-daily
time="2024-03-13T02:31:05Z" level=info msg="new dump" archive=/tmp/kviz-daily-1710297060.gz err="after 0 retries, mongodump log : exit status 1" mlog=/tmp/kviz-daily-1710297060.log plan=kviz-daily
time="2024-03-13T02:31:05Z" level=error msg="BACKUP FAILED: after 0 retries, mongodump log : exit status 1" plan=kviz-daily

Since most of our backups have rentention set to 0 (ie we push to S3, there's no need for local retention), the logs were removed and I've had no chance of figuring out what happened.

For now I've set the retention to 1 (except one very big plan) so if it happens again I can dig through the log. But ideally, no matter the retention levels, we should keep at least the last version of the log. Perhaps simply applying retention+1 to logfiles would suffice in this case (as we don't expect them to be very large anyway).

aneagoe commented 3 months ago

Ok, this is strange. It seems in case of a failure, the logs were removed, even if retention was set to 1 for the failed plan. Not sure why atm.