MongoDB SRV addresses not resolved in Kubernetes

amoscatelli commented 1 year ago

Describe the bug

I initially started a discussion about this, but now I am more convinced is an issue. Anyway since SRV support is an experimental feature, if you tell me to, I'll close this issue right away.

Long story short:

I can't resolve SRV addresses in my kubernetes cluster, both in JVM and Native mode.

Locally everything works in JVM mode. In kubernetes everything works, both in JVM and in Native mode, if I use old format for connection string instead of the SRV one.

Of course, in any of the case described above, I am talking about resolving and connecting to the same MongoDB Atlas environment with the same configuration (as much as possible).

Since using old connection strings works in every mode in my kubernetes, I exclude network/configuration issues with MongoDB Atlas.

Just to be clear this is an old connection string (the one working) : mongodb://quarkus:password@ac-2mpq9bp-shard-00-00.uvyzu7h.mongodb.net:27017,ac-2mpq9bp-shard-00-01.uvyzu7h.mongodb.net:27017,ac-2mpq9bp-shard-00-02.uvyzu7h.mongodb.net:27017/?ssl=true&replicaSet=atlas-7l5e5e-shard-0&authSource=admin&retryWrites=true&w=majority

This is a SRV connection string (the one not working) : mongodb+srv://quarkus:password@quarkus.uvyzu7h.mongodb.net/?retryWrites=true&w=majority

I tried to enabled DEBUG log level to catch something relevant but all I can see, in the end is this error :

2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat12:28:30 INFO traceId=, parentId=, spanId=, sampled= [or.mo.dr.cluster] (cluster-ClusterId{value='6426d1d0c11d2a13b94ccbaa', description='null'}-srv-quarkus.uvyzu7h.mongodb.net) Exception while resolving SRV records: com.mongodb.MongoConfigurationException: Failed looking up SRV record for '_mongodb._tcp.quarkus.uvyzu7h.mongodb.net'.
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.dns.DefaultDnsResolver.resolveHostFromSrvRecords(DefaultDnsResolver.java:92)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.connection.DefaultDnsSrvRecordMonitor$DnsSrvRecordMonitorRunnable.run(DefaultDnsSrvRecordMonitor.java:80)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at java.base@17.0.6/java.lang.Thread.run(Thread.java:833)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchatCaused by: com.mongodb.MongoConfigurationException: Unable to look up SRV record for host _mongodb._tcp.quarkus.uvyzu7h.mongodb.net
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.resolveSrvRequest(MongoDnsClient.java:152)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.getResourceRecordData(MongoDnsClient.java:104)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at com.mongodb.internal.dns.DefaultDnsResolver.resolveHostFromSrvRecords(DefaultDnsResolver.java:74)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat ... 4 more
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchatCaused by: io.smallrye.mutiny.TimeoutException
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:64)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat at io.quarkus.mongodb.runtime.dns.MongoDnsClient.resolveSrvRequest(MongoDnsClient.java:138)
2023-03-31 12:28:30.88 UTCchat-5bbc5746fc-9g6dzchat ... 6 more

I tried many combination of the following relevant properties (but in the end I ALWAYS see the error above) :

quarkus.mongodb.dns.server-host=8.8.8.8 quarkus.mongodb.native.dns.server-host=8.8.8.8 quarkus.mongodb.native.dns.use-vertx-dns-resolver=true quarkus.naming.enable-jndi=true

Since locally everything works, I don't think this is an issue strictly related to MongoDB. Maybe this is more about using both these extensions. Maybe Kubernetes extension should register the quarkus.mongodb.dns.server-host value among the pod dns servers ?

I am sorry I failed to be more concise.

Expected behavior

SRV addresses should be resolved normally

Actual behavior

SRV addresses resolution requests cause a timeout exception

How to Reproduce?

No response

Output of `uname -a` or `ver`

No response

Output of `java -version`

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.0.0 CR1

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

quarkus-bot[bot] commented 1 year ago

/cc @Sgitario (kubernetes), @evanchooly (mongodb), @geoand (kubernetes), @iocanel (kubernetes), @loicmathieu (mongodb)

iocanel commented 1 year ago

I am not familiar with mongo srv records, but somehow using 8.8.8.8 as a dns server and hoping to resolve things running inside your cluster doesn't seem like its ever gonna work.

amoscatelli commented 1 year ago

I tried from an ubuntu helm chart installed in the same namespace, I can succesfully resolve the SRV record just like this :

Also I can tellnet the 27017 ports of my Atlas cluster members, so this exclude further issues with network configuration on Atlas panel. I enabled every ip with a single rule.

amoscatelli commented 1 year ago

The 8.8.8.8 setting was just an attempt. I simply cannot connect to mongodb Atlas out of the box, and everything is working locally OR using the old connection string format.

SRV records are not working on kubernetes only, both in native or JVM mode.

amoscatelli commented 1 year ago

Ok guys I'll close this issue seems this is more tricky than it initially seemed.

Thank you the same

quarkusio / quarkus