stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.63k stars 225 forks source link

Connecting to Mongo #40

Open nikilster opened 10 years ago

nikilster commented 10 years ago

I'm hosting my db on MongoHQ and followed the query string connection as they specified (I'm able to log in through my nodejs - express app). Still getting this error when I try to connect:

/home/ubuntu/.rvm/gems/ruby-2.1.1/gems/mongo-1.10.0.rc1/lib/mongo/functional/authentication.rb:205:in `issue_authentication': Failed to authenticate user 'analytics' on db 'admin'. (Mongo::AuthenticationError)

mongodb://user:pass@candidate.12.mongolayer.com:10280/dbname mongodb://user:pass@candidate.12.mongolayer.com:10280/admin

(tried both and neither work - slightly different errors).

thoughts?

nikilster commented 10 years ago

I'm guess that I can't run the admin commands since its a hosted db - is there any way around this?

nelhage commented 10 years ago

What happens if you try it with the /dbname URL and add --skip-tail? If you don't have admin access to the database you will be able to use the live-streaming feature (which requires access to the replication log), but you should at least be able to do a one-time import.

sadowski commented 10 years ago

Unfortunately, if you look at the code in lib/mosql/streamer.rb line 96 it calls @mongo.get_databases(). This call translates to a listDatabases call on the server, which requires admin access (even if you use --skip-tail).

I'm not really sure why we need to iterate through the databases to find the already specified database. If I've already specified the one I want to connect to, it should just assume that I am connected to the correct database. This might be an area for improvement.

Oh, also, the reason I am experiencing this error is because I'm using a hosted MongoDB solution (MongoHQ) and don't have admin access.

nelhage commented 10 years ago

Yeah, that could be fixed. The reason for that logic was that we use a single collections.yaml schema file for all of our replsets (each replset has a disjoin set of logical databases), and so we needed to discover which databases actually exist in the mongod to sync, instead of blindly connecting to all the databases. I'd happily take a PR that does something more clever.

pshc commented 9 years ago

For the record, the need for admin access can be avoided with --only-db.

AndrewBarba commented 8 years ago

Was anyone able to get this working with Compose (MongoHQ)? If so can you share more details and what options you ended up using? I've tried many times with the --only-db option and that still does not work.

edchan77 commented 8 years ago

I'm using Compose as well, didn't find a solution yet.

edchan77 commented 8 years ago

It works with compose. Looks like I forgot to specify the dbname! --only-db [dbname]

AndrewBarba commented 8 years ago

Hm I did do that, any chance you can post the exact command (without credentials obviously)

AndrewBarba commented 8 years ago

For reference this is what I tried but it fails at the listDatabases command:

mongodb://$USER:$PASSWORD@$HOST:$PORT?replicaSet=set-$REPLICA&readPreference=secondary --only-db $DB
edchan77 commented 8 years ago

@AndrewBarba Perhaps you forgot to specify the the DB after the port?

mosql --mongo mongodb://$USER:$PASSWORD@$HOST:$PORT/$DB?replicaSet=set-$REPLICA&readPreference=secondary --only-db $DB

Btw, you should include both addresses in the replicaset uri, since you don't which will be primary or secondary e.g. mosql --mongo mongodb://user:password@candidate.1.mongolayer.com:1234,candidate.2.mongolayer.com:5678/dbname?replicaSet=set-xxxyyyzzz&readPreference=secondary --only-db dbname