Driver redirecting to internal IP ?

[stefanofornari]

Hello, I have a configuration with two nodes with replication factor 2; they are behind a router with NAT. From the outside, I want to access to one of the two nodes only (for now), therefore I configured the DNS to point to the router and the router NATs it to node 1:

When from the outside I try to access the node, it seems the driver tries to use the internal IP (node1_local_ip) and therefore it fails:

using store jdbc:cassandra://cassandra.mydm.com:9042/uzzstore?localdatacenter=datacenter1
DataStax Java driver for Apache Cassandra(R) (com.datastax.oss:java-driver-core) version 4.17.0
Adding handler to invalidate cached prepared statements on type changes
Using native clock for microsecond precision
[s0|/node1_local_ip:9042  Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.17.0, CLIENT_ID=58789f9d-9a41-4104-af90-5d545a7eea8d}): failed to send request (java.nio.channels.NotYetConnectedException))
Connected to cluster: Uzz, with session: s0
Datacenter: datacenter1; Host: cassandra.mydm.com/<unresolved>:9042; Rack: rack1
Datacenter: datacenter1; Host: /node2_local_ip:9042; Rack: rack1
Node: cassandra.mydm.com/<unresolved>:9042 runs Cassandra v.3.0.8
[s0|/node1_local_ip Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.17.0, CLIENT_ID=58789f9d-9a41-4104-af90-5d545a7eea8d}): failed to send request (java.nio.channels.NotYetConnectedException))

As I see, it tries to use the local ip of the second node

Is that the intended behaviour? Is there any workaround?

ERRATA: the second node is listening to the standard port 9042, not 9142 as in the picture...

[maximevw]

Hello @stefanofornari,

To be honest, I never met such a configuration, so I'm not totally sure of what really happens here.

First of all, the hosts and ports specified in the JDBC URL are simply passed to the underlying Java driver for Apache Cassandra as contact points. These contact points are just used in the specified order to contact one node of the cluster. As soon as a connection is established with a node, this node will be used to discover the other nodes of the cluster (this part is not handled by the JDBC driver).

Here, it seems the connection to the first node was successful because it discovered the cluster "correctly" (we have 2 nodes as expected):

Connected to cluster: Uzz, with session: s0
Datacenter: datacenter1; Host: cassandra.mydm.com/<unresolved>:9042; Rack: rack1
Datacenter: datacenter1; Host: /node2_local_ip:9042; Rack: rack1
Node: cassandra.mydm.com/<unresolved>:9042 runs Cassandra v.3.0.8

If it failed to connect to the first node, you should get an error like this one: com.datastax.oss.driver.api.core.AllNodesFailedException: Could not reach any contact point, make sure you've provided valid addresses.

Here is the code generating the logs above:

this.metadata = this.cSession.getMetadata();

LOG.info("Connected to cluster: {}, with session: {}", Objects.toString(getCatalog(), "<not available>"), this.cSession.getName());
this.metadata.getNodes().forEach(
     (uuid, node) -> LOG.info("Datacenter: {}; Host: {}; Rack: {}", node.getDatacenter(),
      node.getEndPoint().resolve(), node.getRack())
);

As you can see, the displayed hostname is the resolution of the endpoint discovered by the Java driver. So, it's certainly normal the SocketAddress is resolved as /node2_local_ip but I don't know why the port is 9042. Maybe something is misconfigured in your cluster (but not 100% sure).

Also, the communication inter-nodes uses different ports (7000 by default, 7001 with SSL).

As a side note, Cassandra 3.0.x will reach end-of-life very soon (when Cassandra 5.0 will go GA). (source)

[stefanofornari]

Hi @maximevw, thanks for looking into it, I will report the problem to the java driver project.

After further investigation, I realized you are right; the first node is correctly connected and on the right address and port. Then I guess the driver receives the cluster configuration from the node and here is were things break, because it receives the internal IP and not the port. I need to check how I can tell the cluster to advertise each node with proper address and port.

As you can see, the displayed hostname is the resolution of the endpoint discovered by the Java driver. So, it's certainly normal the SocketAddress is resolved as /node2_local_ip but I don't know why the port is 9042. Maybe something is misconfigured in your cluster (but not 100% sure).

I believe the configuration is correct (and it works inside the local network). I think it would be normal for an internal tool, but for a client tool like a driver, it is not normal that internal IPs are disclosed or used.

As a side note, Cassandra 3.0.x will reach end-of-life very soon (when Cassandra 5.0 will go GA)

Thanks for the advice! I am using scylla, so hopefully they will take care of this :)

Many thanks again, I am closing the issue.

[stefanofornari]

BTW, for your information, here is the answer: https://docs.datastax.com/en/developer/java-driver/4.2/manual/core/address_resolution/

[maximevw]

Right, it also means that only the IP address is taken into account to translate the address and probably the port is considered the same for all the nodes of the cluster (I read that here, but I was not sure if this was accurate since it's an old message), if we look at the system.peers table structure.

So, your issue is probably also related to the Cassandra architecture itself and not specifically to the driver behaviour.

[stefanofornari]

Right, it also means that only the IP address is taken into account to translate the address and probably the port is considered the same for all the nodes of the cluster (I read that here, but I was not sure if this was accurate since it's an old message), if we look at the system.peers table structure.

That is hopefully not correct because InetSocketAddress contains the port too. I will check.

[maximevw]

Indeed, the InetSocketAddress contains the port but I have not checked how the driver could determine this and I am skeptical that it can if it relies only on the system.peers table... but maybe it needs more investigation.