search

maxisam / mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
https://maxisam.github.io/mgob/
MIT License
137 stars 19 forks source link

feature request: skip local storage for backups #40

Closed aneagoe closed 1 year ago

aneagoe commented 1 year ago

It would be great to have an option to back-up straight to cloud storage as otherwise, with larger datasets, the local storage requirements get quite big. Alternatively, an option to not keep local backups would also be fine (currently setting retention: 0 does nothing).

maxisam commented 1 year ago

lately I am a bit busy. I will try to do this when I have time. welcome to send a PR

maxisam commented 1 year ago

Hi, could you test this one maxisam/mgob:dev.113

You just need to set retention to 0 in the plan. thx!

aneagoe commented 1 year ago

Will check and report shortly.

aneagoe commented 1 year ago

It doesn't seem to work. The dump is still copied over to /storage after the backup. I double-checked that I'm running the correct image as advised above. If there's any debug information I can gather, please let me know.

maxisam commented 1 year ago

I am so sorry, I think I made a stupid mistake. Could you test dev.115 ?

aneagoe commented 1 year ago

No problem, thanks a lot for taking a look. I've rolled out the new version, but no luck. It's still copying the file over to /storage after the backup is complete. Here's a snippet of the container log:

time="2022-12-23T13:05:08Z" level=info msg="On demand backup started" plan=testdb-daily
time="2022-12-23T13:06:03Z" level=info msg="new dump" archive=/tmp/testdb-daily-1671800708.gz err="<nil>" mlog=/tmp/testdb-daily-1671800708.log
time="2022-12-23T13:06:03Z" level=info msg="Local backup finished Local backup finished `/tmp/testdb-daily-1671800708.gz` -> `/storage/testdb-daily/testdb-daily-1671800708.gz` Duration: 122.00008ms" plan=testdb-daily
time="2022-12-23T13:06:08Z" level=info msg="S3 upload finished " plan=testdb-daily
time="2022-12-23T13:06:08Z" level=info msg="Clean up temp finished Temp folder cleanup finished, `/tmp/testdb-daily-1671800708.gz` is removed." plan=testdb-daily
time="2022-12-23T13:06:08Z" level=info msg="On demand backup finished in 1m0.604096637s archive testdb-daily-1671800708.gz size 270 MB" plan=testdb-daily

I've validated using the correct image, dev.115. Let me know if there's anything else I can do on my side.

maxisam commented 1 year ago

dev.116 should work! I can't believe I made another stupid mistake. This must be the punish from the unit test God.

aneagoe commented 1 year ago

Works like a charm, thanks a lot for this! :tada:

maxisam commented 1 year ago

Thanks for testing! Here is the release https://github.com/maxisam/mgob/releases/tag/1.10.14