search

maxisam / mgob

MongoDB dockerized backup agent. Runs schedule backups with retention, S3 & SFTP upload, notifications, instrumentation with Prometheus and more.
https://maxisam.github.io/mgob/
MIT License
137 stars 19 forks source link

feature request: skip move dumped file from /temp to /storage to save time #65

Closed qingbo-lu closed 1 year ago

qingbo-lu commented 1 year ago

can we skip the mv process(move dumped file from temp to storage dir? it takes quite a long time to mv big dumped file. temp and storage dir are mounted to diffrent disk, mv a huge file consume quite a lot IO. image

qingbo-lu commented 1 year ago

the move can be skipped if local retention is not configured.

qingbo-lu commented 1 year ago

move dumpped file from tmp to storage dir can not be skipped even retention is setted to 0,

maxisam commented 1 year ago

I just wrote a e2e test, it doesn't store to storage path based on the logs

time="2023-09-09T19:11:30Z" level=info msg="mgob dev.172"
time="2023-09-09T19:11:30Z" level=info msg="starting with config: &{LogLevel:info JSONLog:false Host: Port:8090 ConfigPath:/config StoragePath:/storage TmpPath:/tmp DataPath:/data Version:dev.172 UseAwsCli:false HasGpg:false}"
time="2023-09-09T19:11:30Z" level=info msg="mongodump version: 100.6.1 git version: 6d9d341edd33b892a2ded7bae529b0e2a96aae01 Go version: go1.19.3    os: linux    arch: amd64    compiler: gc "
time="2023-09-09T19:11:31Z" level=info msg="NAME:   mc version - manage bucket versioning  USAGE:   mc version COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]  COMMANDS:   enable   enable bucket versioning   suspend  suspend bucket versioning   info     show bucket versioning status    FLAGS:   --config-dir value, -C value  path to configuration folder (default: \"/root/.mc\")   --quiet, -q                   disable progress bar display   --no-color                    disable color theme   --json                        enable JSON lines formatted output   --debug                       enable debug output   --insecure                    disable SSL certificate verification   --limit-upload value          limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited)   --limit-download value        limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited)   --help, -h                    show help    "
time="2023-09-09T19:11:31Z" level=warning msg="aws failed /bin/sh: aws: not found : exit status 127"
time="2023-09-09T19:11:31Z" level=info
time="2023-09-09T19:11:31Z" level=info msg="gpg (GnuPG) 2.2.31 libgcrypt 1.9.4 Copyright (C) 2021 Free Software Foundation, Inc. License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Home: /root/.gnupg Supported algorithms: Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,         CAMELLIA128, CAMELLIA192, CAMELLIA256 Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224 Compression: Uncompressed, ZIP, ZLIB, BZIP2 "
time="2023-09-09T19:11:31Z" level=info msg="Google Cloud SDK 370.0.0 bq 2.0.73 core 2022.01.21 gsutil 5.6 "
time="2023-09-09T19:11:32Z" level=info msg="WARNING: You have 3 updates available. Consider updating your CLI installation with 'az upgrade'  Please let us know how we are doing: https://aka.ms/azureclihats and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy azure-cli                         2.32.0 * "
time="2023-09-09T19:11:32Z" level=info msg="rclone v1.63.1 "
time="2023-09-09T19:11:32Z" level=info msg="Next tmp cleanup run at 2023-09-09 20:00:00 +0000 UTC"
time="2023-09-09T19:11:32Z" level=info msg="New job found, saving to store" plan=mongo-test
time="2023-09-09T19:11:32Z" level=info msg="Next run at 2023-09-09 19:12:00 +0000 UTC" plan=mongo-test
time="2023-09-09T19:11:32Z" level=info msg="New job found, saving to store" plan=no-local
time="2023-09-09T19:11:32Z" level=info msg="Next run at 2023-09-09 19:12:00 +0000 UTC" plan=no-local
time="2023-09-09T19:11:32Z" level=info msg="starting http server on port 8090"
time="2023-09-09T19:12:00Z" level=info msg="Backup started" plan=no-local
time="2023-09-09T19:12:00Z" level=info msg="Backup started" plan=mongo-test
time="2023-09-09T19:12:00Z" level=info msg="Validation: restore backup with : mongorestore --archive=/tmp/mongo-test-1694286720.gz --gzip --host 127.0.0.1 --port 27017 --nsInclude test.* --nsFrom test.* --nsTo test_restore.* "
time="2023-09-09T19:12:00Z" level=info msg="new dump" archive=/tmp/no-local-1694286720.gz err="<nil>" mlog=/tmp/no-local-1694286720.log
time="2023-09-09T19:12:00Z" level=info msg="Validation: connect to mongodb://127.0.0.1:27017"
time="2023-09-09T19:12:00Z" level=info msg="Validation: collection names test2,test"
time="2023-09-09T19:12:00Z" level=info msg="new dump" archive=/tmp/mongo-test-1694286720.gz err="<nil>" mlog=/tmp/mongo-test-1694286720.log
ls: *.gz.encrypted: No such file or directory
time="2023-09-09T19:12:00Z" level=info msg="Local backup finished filename:`/tmp/mongo-test-1694286720.gz`, filepath:`/storage/mongo-test/mongo-test-1694286720.gz`, Duration: 17.157585ms" plan=mongo-test
time="2023-09-09T19:12:00Z" level=info msg="S3 upload finished `/tmp/no-local-1694286720.gz` -> `no-local/backup/no-local-1694286720.gz` Total: 0 B, Transferred: 451 B, Speed: 75.69 KiB/s " plan=no-local
time="2023-09-09T19:12:00Z" level=info msg="Clean up temp finished Temp folder cleanup finished, `/tmp/no-local-1694286720.gz` is removed." plan=no-local
time="2023-09-09T19:12:00Z" level=info msg="Backup finished in 199.655956ms archive no-local-1694286720.gz size 451 B" plan=no-local
time="2023-09-09T19:12:00Z" level=info msg="Next run at 2023-09-09 19:13:00 +0000 UTC" plan=no-local
time="2023-09-09T19:12:00Z" level=info msg="S3 upload finished `/tmp/mongo-test-1694286720.gz` -> `mongo-test/backup/mongo-test-1694286720.gz` Total: 0 B, Transferred: 451 B, Speed: 79.52 KiB/s " plan=mongo-test
time="2023-09-09T19:12:00Z" level=info msg="Clean up temp finished Temp folder cleanup finished, `/tmp/mongo-test-1694286720.gz` is removed." plan=mongo-test
time="2023-09-09T19:12:00Z" level=info msg="Backup finished in 276.078342ms archive mongo-test-1694286720.gz size 451 B" plan=mongo-test
time="2023-09-09T19:12:00Z" level=info msg="Next run at 2023-09-09 19:13:00 +0000 UTC" plan=mongo-test
maxisam commented 1 year ago

You can check the PR #71, the test check the storage folder and the log file. I think it works as expected. Welcome to reopen it if you find any issue