search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.13k stars 4.91k forks source link

Metricbeats cannot connect to mongodb 5.x #29511

Open simllll opened 2 years ago

simllll commented 2 years ago

I've checkd on https://discuss.elastic.co/c/beats, where some posts have been automatically closed already without a solution.

Please include configurations and logs if available.

For confirmed bugs, please report:

simllll commented 2 years ago

I guess it could be related to an "outed" / "old" mongodb driver? But I couldn't find a way to update it locally; i guess it is bundled with Metricbeats? I couldn't find anything related to this issue on the bugs page of the go driver though.. (https://jira.mongodb.org/browse/GODRIVER-1826?jql=project%20%3D%20GODRIVER%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)

elasticmachine commented 2 years ago

Pinging @elastic/integrations (Team:Integrations)

sayden commented 2 years ago

Yes, I think that the driver must be updated.

simllll commented 2 years ago

Any news on this? Thanks

rajsolanki73 commented 2 years ago

any update on this ?

gad26032 commented 2 years ago

Try to use package from mongo website. https://www.mongodb.com/try/download/database-tools

It helped for me.

Little bit details of my case. output from package installed by apt

# mongodump --version
mongodump version: built-without-version-string
git version: built-without-git-spec
Go version: go1.10.1
   os: linux
   arch: amd64
   compiler: gc
OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017

and mongodump fails all the time with the following output

Failed: error connecting to db server: server returned error on SASL authentication step: BSON field 'saslContinue.mechanism' is an unknown field.

After I installed the version from mongo website it run with success and mongodump version is looks right now.

# mongodump --version
mongodump version: 100.5.2
git version: e2842eb549308c16e070fb2e197d4631cfd085cb
Go version: go1.16.3
   os: linux
   arch: amd64
   compiler: gc

Thanks to: https://stackoverflow.com/questions/68526162/mongodb-connection-error-sasl-with-docker-windows-10 it it made me think.

simllll commented 2 years ago

hi @gad26032, thanks for commeting, but the issue is not the with mongodb toolset (mongodump), but with the metricbeats client which has a mongodb client built in. mongodump and other mongo tools are working on my side

gad26032 commented 2 years ago

hi @gad26032, thanks for commeting, but the issue is not the with mongodb toolset (mongodump), but with the metricbeats client which has a mongodb client built in. mongodump and other mongo tools are working on my side

Maybe it is something that has the same cause. Maybe metcbeats use mongo-tools binaries, and it leads to the same error that I had. Have you tried to reinstall mongo-tools? from the official mongo site?

sayden commented 2 years ago

Beats still uses the old https://github.com/go-mgo/mgo library to connect to MongoDB, which is currently unmaintained.

Beats must switch the library to the official https://github.com/mongodb/mongo-go-driver which AFAIK wasn't available when the module was incorporated into the codebase.

The structure of the events that Beats generates must be maintained, of course or do the minimum possible amount of changes. So, for example, mongodb.dbstats.avg_obj_size.bytes must still be written in the same JSON path after switching libraries.

@rameshelastic FYI

JohnKnoepfle commented 2 years ago

Just curious if we have an update on this issue. Some users are impacted by this.

cyberjeff-u8t commented 2 years ago

We are impacted by this issue since we migrate our database from 4.x to 5.x Any estimate please ?

sayden commented 2 years ago

We have not received any external contributions to fix the issue and the capacity of the team at Elastic is limited, but it's in the radar of the team.

EdCSM commented 2 years ago

@sayden Any update / ETA? Some users are planning to upgrade MongoDB and this bug is a blocker.

sayden commented 2 years ago

Right now, there's a plan to release this in some minor version ASAP. It's inconvenient, I feel your pain, but the fix is coming :wink:

rajsolanki73 commented 2 years ago

any update on this ?

sayden commented 2 years ago

There's a PR ongoing already :slightly_smiling_face: https://github.com/elastic/beats/pull/31185/

rajsolanki73 commented 2 years ago

sorry if i m asking elementary question.

1) when will this be available via metricbeat release or upgrade ? 2) can I try this PR ? if yes then how ?

cyberjeff-u8t commented 2 years ago

Hi, Same question as @rajsolanki73. Could someone answer?

rajsolanki73 commented 2 years ago

any update on this ?

cyberjeff-u8t commented 2 years ago

any update on this ?

it is now available and working with metricbeat 8.3.1 !

rajsolanki73 commented 2 years ago

well for me if i add single server ( i have 3 server cluster with replica set in following template it works.

hosts: ["mymongoserver1:27017"] username: myuser password: mypassword

but if add multiple hosts

hosts: ["mymongoserver1:27017",” mymongoserver2:27017”,” mymongoserver3:27017”] username: myuser password: mypassword

I get following error.

{"log.level":"error","@timestamp":"2022-07-18T08:21:47.507-0500","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset mongodb.status: could not create mongodb client: could not create mongodb client: a direct connection cannot be made if multiple hosts are specified","service.name":"metricbeat","ecs.version":"1.6.0"}

So then i tried following template

hosts: ["mongodb://myuser:mypass@mymongoserver1:27017", " mongodb://myuser:mypass@mymongoserver2:27017", mongodb://myuser:mypass@mymongoserver3:27017"]

{"log.level":"error","@timestamp":"2022-07-18T08:33:34.818-0500","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset mongodb.status: failed to retrieve 'serverStatus': server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: myuser:mypass@mymongoserver1:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: address mongodb:// myuser:mypass@mymongoserver1:27017: too many colons in address }, ] }","service.name":"metricbeat","ecs.version":"1.6.0"}

asher-lab commented 2 years ago

I am facing the same issue with Mongo 6

ssingh13-rms commented 1 year ago

While setting up Metricbeats to monitor MongoDB, I too faced the error too many colons in address. I followed the standard MongoDB url format (which the mongodb.yml alluded to as well) but it failed to work.

I was able to setup the connection by changing the config like so

  hosts: ["<your_host>:<port>"]
  username: <your_username>
  password: <your_password>
TomonoriSoejima commented 1 year ago

I also faced the same and my environment is below.

It looks like metricbeat 8.8.1 is having an error on connecting to even older versions such as above.

andrei-gutierrez commented 1 year ago

any idea when this will be fixed?

algo7 commented 11 months ago

any updates?

Amaimersion commented 7 months ago

I had too many colons in address error while connecting to MongoDB 7. Then in Metricbeat config I changed hosts: ["mongodb://localhost:27017"] to hosts: ["localhost:27017"] and it worked.