percona / mongodb_exporter

A Prometheus exporter for MongoDB including sharding, replication and storage engines
Apache License 2.0
1.18k stars 423 forks source link

Connection Errors #543

Closed pseymournutanix closed 4 months ago

pseymournutanix commented 2 years ago

Cannot Connect the Exporter to the Instance. If I use mongosh all seems OK:-

$ mongosh "mongodb://<user>:<pass>@<instance>.us-west-2.aws:27017/bfadb?ssl=true"                                                                                                                                                                                                             
Current Mongosh Log ID: 62fa0606444233a5cd08bde8
Connecting to:      mongodb://<credentials>@<instance>.us-west-2.aws:27017/bfadb?ssl=true&directConnection=true&appName=mongosh+1.5.4
Using MongoDB:      4.2.15
Using Mongosh:      1.5.4

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

[direct: mongos] bfadb>

However various different combinations of trying to connect the exporter all fail:-

$ ./mongodb_exporter --mongodb.uri="mongodb://<user>:<pass>@<instance>.us-west-2.aws:27017/bfadb?ssl=true" --mongodb.direct-connect --log.level="debug" 
DEBU[0000] Compatible mode: true
DEBU[0000] Connection URI: mongodb://<user>:<pass>@<instance>.us-west-2.aws:27017/bfadb?ssl=true
level=info ts=2022-08-15T08:43:38.148Z caller=tls_config.go:195 msg="TLS is disabled." http2=false
ERRO[0003] Cannot connect to MongoDB: cannot connect to MongoDB: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-1": (AuthenticationFailed) Authentication failed.

I think I have tried every combination I can think off they all come back with the above error or panic with:-

RRO[0020] Cannot connect to MongoDB: cannot connect to MongoDB: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: <instance>.us-west-2.aws:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: connection(<instance>.us-west-2.aws:27017[-64]) incomplete read of message header: read tcp 10.66.20.119:58320->10.192.13.17:27017: read: connection reset by peer }, ] }
2022/08/15 09:35:41 http: panic serving 127.0.0.1:58278: runtime error: invalid memory address or nil pointer dereference
goroutine 127 [running]:
net/http.(*conn).serve.func1()
    /opt/hostedtoolcache/go/1.17.12/x64/src/net/http/server.go:1802 +0xb9
panic({0x1732c20, 0x1e65210})
a-gavin commented 2 years ago

I don't know that it's necessarily the same issue, but the error message you posted looks familiar.

I was having some authentication issues caused by a MongoDB goland driver workaround to support passwords which contain "+". More details in the following issue: #534

The main similarity I see between the issue I had and the issue you're having is you are using a non-admin authSource, something I am attempting to fix in an open PR: #535

pseymournutanix commented 2 years ago

Hi,

Thanks for looking. The creds don't have special characters in them.

I tried building from your branch and I got this error:-

error occurred during connection handshake: x509: “mongo-0” certificate is not standards compliant

Which is probably valid as the cert is self-signed.

I tried adding tlsAllowInvalidCertificates=true into the connection URI but no joy.

Appreciate the help.

Cheers

pseymournutanix commented 2 years ago

I added TLS stuff and now it's working - well making a connection and getting some data back.

I don't get a cluster label (which the dashboard expect), nor do I get a full metric set the context deadline exceeded comes up a lot.

Anyway thats another issue.

a-gavin commented 2 years ago

One error to the next haha! If you run in to TLS cert errors again, definitely recommend following this section of the MongoDB docs for setting up testing TLS certs.

Do you mind posting what fixed this issue for you so others might benefit? Also, did you build with Percona's repo for this solved approach?

mbeckDWRE commented 2 years ago

We get the same error when connecting to an instance of mongo we set up with the community operator:

"Cannot connect to MongoDB: cannot connect to MongoDB: connection() error occured during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."

We run the exporter as a sidecar and tried various connection strings: --mongodb.uri=mongodb://${MONGODB_USER}:${MONGODB_PW}@localhost:27017/admin?ssl=false

Password contains special characters which are URL encoded and the mongoshell login is working fine

 mongo mongodb://${MONGODB_USER}:${MONGODB_PW}@localhost:27017/admin?ssl=false
MongoDB shell version v4.4.10
connecting to: mongodb://localhost:27017/admin?compressors=disabled&gssapiServiceName=mongodb&ssl=false
Implicit session: session { "id" : UUID("1c5e3571-fb7b-4b61-8143-a9a89a24b4ce") }
MongoDB server version: 4.4.10
Welcome to the MongoDB shell.

Connecting to the service url instead of a dedicated instance works but is obviously not what we want for a sidecar: mongodb+srv://${USER}:${PW}@mongodb-svc.mongodb.svc.cluster.local/admin?replicaSet=mongodb-platform&ssl=false

Tried several combinations of connection url and exporter versions (0.31.0 -0.34.0) and would appreciate any help.