search

rwynn / monstache

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
https://rwynn.github.io/monstache-site/
MIT License
1.28k stars 180 forks source link

Monstache Atlas SRV connection issue #428

Closed aspire-shatheesh closed 3 years ago

aspire-shatheesh commented 4 years ago

Hi @rwynn

First of all thanks for this wonderful tool. We are able to use it on large datasets (30M+) without any issue. We are in the process of moving from standalone mongo on ec2 box to managed services such as atlas. So while analyzing it, I created a free account in mongo db atlas and tried to set up a monstache sync. The connection string has srv record format and when i try to connect to it, it is throwing the below error

Running a non docker version, i.e direct linux build and the toml file has just connection information alone Env: ubuntu 18.04 personal machine.

./build/linux-amd64/monstache -verbose -f CONFIG-srv.toml
ERROR 2020/09/04 08:26:59 Unable to connect to MongoDB using URL mongodb+srv://REDACTED@cluster0.xxxx.mongodb.net/?retryWrites=true&w=majority: error parsing uri: lookup cluster0.xxxx.mongodb.net on 127.0.0.53:53: cannot unmarshal DNS message

But when I connect with old connection string format as suggested here i.e non SRV it works as expected.

Below is the log on successful connection and also info on the versions of tools used

INFO 2020/09/04 02:55:14 Started monstache version 6.2.2
INFO 2020/09/04 02:55:14 MongoDB go driver v1.1.2
INFO 2020/09/04 02:55:14 Elasticsearch go driver 7.0.6
INFO 2020/09/04 02:55:14 Successfully connected to MongoDB version 4.2.8

So the question is, is it something that monstache works only with non-srv connection string formats?

Looks like the documentation link mentioned here doesn't exist.

rwynn commented 4 years ago

@aspire-shatheesh can you try with monstache 6.7.0. That would upgrade the go MongoDB driver to 1.3.5 I believe. Not sure if that would fix it but worth trying.

rwynn commented 4 years ago

I was able to find some issues related to this

aspire-shatheesh commented 4 years ago

Hi @rwynn

Looks like the issue still persists

ERROR 2020/09/04 20:48:53 Unable to connect to MongoDB using URL mongodb+srv://REDACTED@cluster0.xxxxx.mongodb.net/?retryWrites=true&w=majority: error parsing uri: lookup cluster0.xxx.mongodb.net on 127.0.0.53:53: cannot unmarshal DNS message

whereas the non srv works fine. Looks like something to do with go driver


INFO 2020/09/04 20:48:41 Started monstache version 6.7.0
INFO 2020/09/04 20:48:41 Go version go1.14.1
INFO 2020/09/04 20:48:41 MongoDB go driver v1.3.5
INFO 2020/09/04 20:48:41 Elasticsearch go driver 7.0.18
INFO 2020/09/04 20:48:41 Successfully connected to MongoDB version 4.2.8
rwynn commented 4 years ago

@aspire-shatheesh thanks for testing and confirming the error still exists in 6.7.0. I think the root cause may be even lower than the driver as the MongoDB engineers are looking for a fix at the go level.

aspire-shatheesh commented 4 years ago

okay thanks @rwynn for confirming. Please close this ticket if you feel this is beyond the scope of our tool