Open alwynpan opened 1 year ago
Hi there this has been added to version 3.4.0
. Changelog - Please checkout the README to understand how to change it to atlas mode and also the URI override that has been built in. Let me know if you encounter any issues.
Hi,
Having some issues using this new version.
When specifying MONGO_HOST_TYPE=atlas
and setting DB_HOST=clustername.id.mongodb.net
, the code from line 379 is preventing the backup from running, since you are trying to resolve a SRV record with nc
and it is failing:
Mongo Host 'clustername.id.mongodb.net' is not accessible, retrying.. (5 seconds so far)
Thanks for the report. 3.4.2
should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.
Thanks for the report.
3.4.2
should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.
Thanks! I'll try it :)
Thanks for the report.
3.4.2
should resolve this. I have stripped the host availbility check when backing up with a `mongo+srv+ prefix.
It wasn't working, so I checked the code again and found: --uri="${MONGO_URI_PREFIX}${DB_HOST}:${DB_PORT}
.
When using an SRV record, the port shouldn't be specified, just tried it in a local mongodump to be sure:
In my opinion, it would be nice to have a way, through ENV vars, to specify the full uri just like mongodb+srv://username:password@cluster.id.mongodb.net
instead of building it from multiple vars like host, user, password, etc. Maybe check if MONGO_CUSTOM_URI is specified and use it instead of building this: --uri="${MONGO_URI_PREFIX}${DB_HOST}:${DB_PORT}" ${MONGO_USER_STR}${MONGO_PASS_STR}${MONGO_AUTH_STR}${MONGO_DB_STR}
.
I agree with your opinion. I can do that. Help me understand, if you send a username and password in the URI - that would negate having to use the Mongo User and Pass variables I have following the URI? A few real world examples would help and I can certainly build something in.
if you send a username and password in the URI - that would negate having to use the Mongo User and Pass variables I have following the URI?
Yes. The uri can contain the username and password in the format that I mentioned before mongodb+srv://username:password@cluster.id.mongodb.net/defaultDb?options=extraStuff
Here is an example of an uri from mongo compass:
Basically if you accept an URI from the env vars (don't modify it) and pass it through to mongodump, it would allow the user to access any mongodb instance (self-hosted or atlas) and even specify extra options. For most of us would be the best way to handle mongodb connections, since we use atlas with SRV records and the URI is automatically handled to use by atlas.
Perfect, thanks for the input here. Let me give this a think over the next 24 hours and come up with a way forward.
@tiredofit @Dinip, my understanding is we should be able to use mongodb+srv://cluster.id.mongodb.net
with the --uri
parameter and specify the username
and password
etc in separate parameters.
I am sorry, I created this issue and planned to implement this myself, but I was dragged into some urgent work and didn't have a chance to do that. Once I finished the urgent job (hopefully by tomorrow), I can help to do testing and debugging.
In https://github.com/tiredofit/docker-db-backup/commit/a377f570f16716b553d1493c44d7a78bd70e5226 I've added support for a MONGO_CUSTOM_URI
environment variable. I had to do some parsing of the URI string for filename generation, but also left the parsed variables in place for other needs if ever down the road.
If you aren't happy with the DB_NAME
or DB_HOST
of the generated file you can override it.
Instead of pushing a bad release - I've switched over to tiredofit/db-backup:develop
if you were willing to pull and test.
Something must be wrong with this validation if [ "${MONGO_HOST_TYPE,,}" != "atlas" ] || [ -n "${MONGO_CUSTOM_URI}" ]; then
My env vars:
environment:
- DB_TYPE=mongo
- MONGO_CUSTOM_URI=mongodb+srv://user_redacted:pass_redacted@cluster.redacted.mongodb.net
Thanks for feedback. Pushing new develop
with proper variable check.
Pulled the new docker image and seems to be doing the same 🤔
That's bizarre, I had switched it to this:
if [ "${MONGO_HOST_TYPE,,}" != "atlas" ] || [ -z "${MONGO_CUSTOM_URI}" ]
For time being I am going to strip any host checking just to get us to the good stuff.
The backup now works but is having some errors at the end and it does not upload to the s3 bucket.
Timeline: At the beginning
[tiredofit/db-backup:develop 19:40:24 /] $ backup-now
** Performing Manual Backup
/assets/functions/10-db-backup: line 462: syntax error near unexpected token `fi'
/assets/functions/10-db-backup: line 462: ` fi'
/etc/services.available/10-db-backup/run: line 56: check_availability: command not found
Reads the data from the mongo instance and writes it to the archive.gz file
At the end
/assets/functions/10-db-backup: line 187: check_exit_code: command not found
/assets/functions/10-db-backup: line 188: generate_checksum: command not found
/assets/functions/10-db-backup: line 189: move_dbbackup: command not found
/assets/functions/10-db-backup: line 190: post_dbbackup: command not found
2022-09-20.19:41:46 [INFO] ** [db-backup] Backup routines finish time: 2022-09-20 19:41:46 GMT with overall exit code 0
2022-09-20.19:41:46 [NOTICE] ** [db-backup] Backup routines time taken: Hours: 0 Minutes: 01 Seconds: 19
/etc/services.available/10-db-backup/run: line 79: cleanup_old_data: command not found
Doesn't upload to S3
Thanks, looking into and will have a fix momentarily.
Right - Next :develop
will fix and work :)
Yep, now seems to work! Thanks for the update 😀 If you need help testing after you re-enable the host verification, let me know!
Awesome. I'm going to do that one right now - If you wanted to check the new :develop
and we pass we should be good to move this to master and have a new release.
something is weird with that verification 🤔
Interesting. Can I get some of the environment variables you are using? I can't seem to recreate this whatsoever.
Not much of a bash expert but just done some research and seems like -z "$variable"
checks if the variable is unset or is set to an empty string. In this case, $MONGO_CUSTOM_URI
is defined, so it would return false.
For example, in line 68
if [ -z "${MONGO_CUSTOM_URI}" ] ; then
[[ ( -n "${DB_USER}" ) ]] && MONGO_USER_STR=" --username ${DB_USER}"
[[ ( -n "${DB_PASS}" ) ]] && MONGO_PASS_STR=" --password ${DB_PASS}"
[[ ( -n "${DB_NAME}" ) ]] && MONGO_DB_STR=" --db ${DB_NAME}"
[[ ( -n "${DB_AUTH}" ) ]] && MONGO_AUTH_STR=" --authenticationDatabase ${DB_AUTH}"
else
your are checking if custom uri is empty to set the values to the other variables.
With this check
if [ "${MONGO_HOST_TYPE,,}" = "atlas" ] || [ -z "${MONGO_CUSTOM_URI}" ]; then
print_debug "Skipping Connectivity Check"
else
you are checking if either the host type is atlas or the custom uri is empty to skip the check, but the check for custom uri will return false since it is set.
Made some tests:
But going back to the check that you had before [ -n "${MONGO_CUSTOM_URI}" ]
seems to work
New :develop
is ready - I stripped the ||
OR Statement as I'm clearly not capable this week of building those :)
Well...
Here if [ -n "${MONGO_CUSTOM_URI}" ]; then MONGO_DB_TYPE="atlas" ; fi
you are setting MONGO_DB_TYPE
but it should be MONGO_HOST_TYPE
:)
Death by environment variables - New :develop
being built!
Description of the feature Currently
docker-db-backup
does not support MongoDB Atlas. It would be great to supportmongodb+srv://
type of hosts. The simplest way is to use--uri
instead of--host
withmongodump
.Benefits of feature By replacing
--host
with--uri
and addingDB_HOST_TYPE
(default tomongodb
and can be set tomongodb+srv
), it makes themongodump
more flexible.