search

MARTIMM / raku-mongodb-driver

MongoDB driver for Raku
Artistic License 2.0
18 stars 8 forks source link

Add support for mongodb+srv connection string #34

Open Zer0-Tolerance opened 6 months ago

Zer0-Tolerance commented 6 months ago

Hi Marcel, I'm trying to connect to MongoDB Atlas instance but it's required to use a connection string like this one: Parsing error in url 'mongodb+srv://user:password@host.o9jgs.mongodb.net/?retryWrites=true&w=majority' but the current driver don'ts know how to handle this connection and complains with this: 2024-02-25 19:11 UTC 58.886576 [F][1][Uri][380]: Parsing error in url 'mongodb+srv://host.o9jgs.mongodb.net/?retryWrites=true&w=majority' Would you be able to add support for this ? Or tell me how I can workaround this ?

Thanks a lot.

MARTIMM commented 6 months ago

the 'mongodb+srv' part throws the problem. I may be able to add a rule for it but can not test if there will be another problem caused by it. Also, will there be more extensions to take care of? Anyways I try to fix it and see what happens.

MARTIMM commented 6 months ago

I've published a new version 0.44.0. Must be available soon. Let me know if it works.

Zer0-Tolerance commented 6 months ago

Thanks a lot I'll do the testing today. But I think you need to handle the SSL connection as well because it is required by MongoDB Atlas.

the 'mongodb+srv' part throws the problem. I may be able to add a rule for it but can not test if there will be another problem caused by it. Also, will there be more extensions to take care of? Anyways I try to fix it and see what happens.

Zer0-Tolerance commented 6 months ago

btw you can also do some testing with the free tier MongoDB Atlas, the typical URI looks like this: mongodb+srv://<user>:<password>@host.o9jgs.mongodb.net/?retryWrites=true&w=majority&appName=yourcluster-cluster and from what I've been reading to far it is discovering the real servers with a DNS SRV request + a DNS TXT request to get some parameter for the connection. more info here

Zer0-Tolerance commented 6 months ago

First feedback on the new version, it doesn't seems to connect successfully. This is probably due to the lack of SSL / TLS support ?

MARTIMM commented 6 months ago

I do not have it implemented so that is a problem. Don't know how to do that in a short time. Need to investigate.

MARTIMM commented 6 months ago

I will do it slowly, step by step.

I also need to attend to two other projects though.

MARTIMM commented 5 months ago

Small update..... Done the above and some checking on URI options but I am now stuck for some time, setting up a connection to a server. The server starts properly using some certificate and key but I cannot convince OpenSSL to do the work I need. Errors like 'Private key does not match the public certificate' from OpenSSL and 'No SSL certificate provided by peer; connection rejected' on the server.

Zer0-Tolerance commented 5 months ago

Hi Marcel, thanks for the update, I didn't know Mongosh was doing some kind of certificate pinning. I'll try to do some more testing this week-end.

MARTIMM commented 5 months ago

hi @Zer0-Tolerance, at the moment I was doing some other work on other distros. It helps to empty my mind by doing something else before returning to this task.

MARTIMM commented 4 months ago

Taraaaah!. There is hope! My little test works. Now I only need to understand it. Of course, it must then be coded into the driver......

Zer0-Tolerance commented 4 months ago

Hi Marcel, I'm glad to hear that.

MARTIMM commented 3 months ago

I have pushed version 0.45.0 today. The URI accepts the tls, tlsCAFile, and tlsCertificateKeyFile options. I have tested it with homemade certificates so I am curious if you get it running with proper certificates and keys.

Zer0-Tolerance commented 3 months ago

Hi , this is great ! I'll test the new version asap.

Zer0-Tolerance commented 3 months ago

Hi Marcel, Just did the test, It fails with this error

$dburl="mongodb+srv://<user>:<password>@host.o9jgs.mongodb.net/?retryWrites=true&w=majority&appName=yourcluster-cluster";
--------------------------------------------------------------------------------
2024-05-24 22:30 UTC

43.664272 [F][1][Uri][516]: Found server 'test-dev-cluster-shard-00-00.o9jgs.mongodb.net' must be in same domain 'test-dev-cluster.o9jgs.mongodb.net'
Found server 'test-dev-cluster-shard-00-00.o9jgs.mongodb.net' must be in same domain 'test-dev-cluster.o9jgs.mongodb.net'
  in sub fatal-message at /Users/.rakubrew/versions/moar-2024.03/share/perl6/site/sources/02D61B5AC84C560843B1B47BFD76B5376184893F (MongoDB::Log) line 331
  in method get-srv-hosts at /Users/.rakubrew/versions/moar-2024.03/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 516
  in block  at /Users//.rakubrew/versions/moar-2024.03/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 290
  in submethod BUILD at /Users/.rakubrew/versions/moar-2024.03/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 280
  in submethod BUILD at /Users/.rakubrew/versions/moar-2024.03/share/perl6/site/sources/63FF6B179C583222B22CF1D14143D549E357423A (MongoDB::Client) line 205
  in method new at /Users//.rakubrew/versions/moar-2024.03/share/perl6/site/sources/63FF6B179C583222B22CF1D14143D549E357423A (MongoDB::Client) line 137
  in block <unit> at <unknown file> line 1
  in any <main> at /Users//.rakubrew/versions/moar-2024.03/bin/../share/perl6/runtime/perl6.moarvm line 1
  in any <entry> at /Users//.rakubrew/versions/moar-2024.03/bin/../share/perl6/runtime/perl6.moarvm line 1
MARTIMM commented 3 months ago

I do not see any certificate / key specs in your URI. Does it work like that? If so, I need to find a way to have more than one domain to test such a situation. I only have a 'localhost'.

MARTIMM commented 3 months ago

Shame on me, I looked in the wrong spot asking Google to find errors it could not possibly find. It was a test in the Uri class throwing this error. Should have looked better :-|.

I have repaired the problem but I cannot try this out myself. I will publish this version, 0.45.1 and hopefully, this one will work for you.

Zer0-Tolerance commented 2 months ago

Hi, there is a different error now :

--------------------------------------------------------------------------------
2024-06-15 23:55 UTC

39.752469 [F][1][Uri][516]: Found server 'test-dev-cluster-shard-00-01.o9jgs.mongodb.net' must be in same domain 'test-dev-cluster.o9jgs.mongodb.net'
Found server 'test-dev-cluster-shard-00-01.o9jgs.mongodb.net' must be in same domain 'test-dev-cluster.o9jgs.mongodb.net'
  in sub fatal-message at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/02D61B5AC84C560843B1B47BFD76B5376184893F (MongoDB::Log) line 331
  in method get-srv-hosts at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 516
  in block  at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 290
  in submethod BUILD at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/A52B53AB41F36AF34A71F1BC5E4ECE5BFDA68301 (MongoDB::Uri) line 280
  in submethod BUILD at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/63FF6B179C583222B22CF1D14143D549E357423A (MongoDB::Client) line 205
  in method new at /Users/.rakubrew/versions/moar-2024.05/share/perl6/site/sources/63FF6B179C583222B22CF1D14143D549E357423A (MongoDB::Client) line 137
  in block <unit> at mongop5.raku line 40

to connect to the cluster with mongosh I just do this so I don't need to provide any cert or CA: mongosh "mongodb+srv://test-dev-cluster.o9jgs.mongodb.net/" --apiVersion 1 --username admin

MARTIMM commented 2 months ago

I am not sure but I get the feeling that you are not running the latest version. It should be version 0.45.1. The error above is from a previous version.

Furthermore, I assume the atlas server is started with the server TLS option disabled. However, the raku modules will automatically turn on TLS when srv records are used. So, when the encrypted connection isn't needed you have to explicitly turn TLS off … tls=false … in the URI.

Zer0-Tolerance commented 2 months ago

same error with the version 0.45.1 and tls=false

MARTIMM commented 2 months ago

Sorry that it still fails, I'll look into it

MARTIMM commented 2 months ago

I clipped too little from the owner domain (of the SRV records) where the server would be in. should have been o9jgs.mongodb.net instead of test-dev-cluster.o9jgs.mongodb.net. So, version 0.45.2 is in the making and perhaps available later on. I do not know how long it takes for zef to find in from CPAN.

Zer0-Tolerance commented 2 months ago

Quick update with latest version 0.45.2, new error now:

2024-06-22 23:07 UTC
42.885552 [F][1][Uri][538]: Only one TXT record is accepted for this domain 'test-dev-cluster.o9jgs.mongodb.net'
MARTIMM commented 2 months ago

I think I got this test right, but I'm curious about which TXT records there are for the specific domain. Could you please send these to me?

According to mongodb:

Like the standard connection string, the DNS seed list connection string supports specifying options as a query string. With a DNS seed list connection string, you can also specify the following options via a TXT record:

You may only specify one TXT record per mongod instance. If multiple TXT records appear in the DNS and/or if the TXT record contains an option other than replicaSet or authSource, the client will return an error.

You can find this text in the connection-string reference.

MARTIMM commented 2 months ago

Any luck? You could try dig test-dev-cluster.o9jgs.mongodb.net TXT.

Zer0-Tolerance commented 2 months ago

Hi, please find the output from dig darkrecon-dev-cluster.o9jgs.mongodb.net TXT:

; <<>> DiG 9.10.6 <<>> darkrecon-dev-cluster.o9jgs.mongodb.net TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18451
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;darkrecon-dev-cluster.o9jgs.mongodb.net. IN TXT

;; ANSWER SECTION:
darkrecon-dev-cluster.o9jgs.mongodb.net. 60 IN TXT "authSource=admin&replicaSet=DarkRecon-DEV-cluster-shard-0"

;; AUTHORITY SECTION:
mongodb.net.        172710  IN  NS  ns-1647.awsdns-13.co.uk.
mongodb.net.        172710  IN  NS  ns-1043.awsdns-02.org.
mongodb.net.        172710  IN  NS  ns-761.awsdns-31.net.
mongodb.net.        172710  IN  NS  ns-430.awsdns-53.com.

;; ADDITIONAL SECTION:
ns-761.awsdns-31.net.   5520    IN  A   205.251.194.249

;; Query time: 22 msec
;; SERVER: 10.4.0.1#53(10.4.0.1)
;; WHEN: Sat Jul 06 10:42:40 CEST 2024
;; MSG SIZE  rcvd: 291
MARTIMM commented 2 months ago

Fixed a minor something, caused by copy / paste code. It is a pity that I can not use a server of yours safely to test things more fully. Now it is a small step at a time until the next bug is found. Sorry for the inconvenience.

MARTIMM commented 2 months ago

Fixed a minor bug caused by copy/paste of code. It is a pity that I cannot use a server of yours safely so I can test everything more fully.