Cannot resolve domain in client in k8s

[z0mb1ek]

I has search domain at my /etc/resolv.conf:

cat /etc/resolv.conf 
search dev.svc.cluster.local svc.cluster.local cluster.local local
nameserver 10.222.0.10
options ndots:5

On 2.16.8 version it worked well, i resolved nats:4222 on my service in kubernetes. But when i upgrade to 2.16.9 it starts raising exception:

exceptionOccurred, Exception: java.io.IOException: java.lang.IllegalArgumentException: port out of range:-1

i see this commits https://github.com/nats-io/nats.java/pull/847/files

could that be the reason?

[scottf]

Hostname resolution was added, it can be turned off via Options.Builder().noResolveHostnames()

Do you happen to have a stacktrace for that exception, maybe we can find a way to address it.

[z0mb1ek]

Yes, with this option work well now. Please explain why it is needed?

Caused by: java.io.IOException: Unable to connect to NATS servers: [nats://nats:4222]
    at io.nats.client.impl.NatsConnection.connect(NatsConnection.java:240)
    at io.nats.client.impl.NatsImpl.createConnection(NatsImpl.java:29)
    at io.nats.client.Nats.createConnection(Nats.java:303)
    at io.nats.client.Nats.connect(Nats.java:210)

there are no more logs

[scottf]

It's needed because hostname resolution was added and made as the default. It is possible that there is a way for resolution to work properly, which is why I needed the stack trace for your original configuration. Maybe you can extend then override ErrorListenerLoggerImpl and print the stack trace instead of the default which is this:

    public void exceptionOccurred(final Connection conn, final Exception exp) {
        LOGGER.severe(() -> supplyMessage("exceptionOccurred", conn, null, null, "Exception: ", exp));
    }

[z0mb1ek]

here they are:

java.io.IOException: java.lang.IllegalArgumentException: port out of range:-1
    at io.nats.client.impl.SocketDataPort.connect(SocketDataPort.java:99)
    at io.nats.client.impl.SocketDataPort.connect(SocketDataPort.java:52)
    at io.nats.client.impl.NatsConnection.tryToConnect(NatsConnection.java:421)
    at io.nats.client.impl.NatsConnection.connect(NatsConnection.java:203)
    at io.nats.client.impl.NatsImpl.createConnection(NatsImpl.java:29)
    at io.nats.client.Nats.createConnection(Nats.java:303)
    at io.nats.client.Nats.connect(Nats.java:210)
    at tech.livecom.streams.config.NatsConfig.testBean(NatsConfig.kt:110)
    at tech.livecom.streams.config.NatsConfig$$SpringCGLIB$$0.CGLIB$testBean$2(<generated>)
    at tech.livecom.streams.config.NatsConfig$$SpringCGLIB$$2.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
    at tech.livecom.streams.config.NatsConfig$$SpringCGLIB$$0.testBean(<generated>)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:491)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
    at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:66)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293)
    at tech.livecom.streams.ApplicationKt.main(Application.kt:15)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.IllegalArgumentException: port out of range:-1
    at java.base/java.net.InetSocketAddress.checkPort(InetSocketAddress.java:152)
    at java.base/java.net.InetSocketAddress.<init>(InetSocketAddress.java:233)
    at io.nats.client.impl.SocketDataPort.connect(SocketDataPort.java:77)
    ... 41 more

[scottf]

So that didn't help. I guess I need to talk to our Kubernetes guy to figure out how exactly ip addresses are resolved and why it doesn't fail. I guess for now I can try to determine if port is -1 after resolve and toss that resolution.

[scottf]

If I build a snapshot from a branch, would you be able to test? I'll add some debug to the hostname resolution and we'll see what exactly the resolver does.

[z0mb1ek]

yes, i can test it

[scottf]

Can you try this:

import io.nats.client.Connection;
import io.nats.client.Nats;
import io.nats.client.Options;
import io.nats.client.api.ServerInfo;
import io.nats.client.support.NatsUri;

import java.net.InetAddress;
import java.net.URISyntaxException;
import java.net.UnknownHostException;

public class TestResolve {
    public static void main(String[] args) {

        // simple resolve
        try {
            NatsUri nuri = new NatsUri("connect.ngs.global");
            System.out.println(nuri);
            System.out.println("OLD: " + nuri.reHost("10.111.7.11://10.111.7.11:4222"));
            System.out.println("NEW: " + reHost(nuri, "10.111.7.11://10.111.7.11:4222"));
            resolve(nuri);
        } catch (URISyntaxException e) {
            System.out.println(e);
        }

        // resolve based on server discovery from actual connection
        Options options = new Options.Builder()
            .server(Options.DEFAULT_URL)
            .build();

        try (Connection nc = Nats.connect(options)) {
            ServerInfo si = nc.getServerInfo();
            for (String url : si.getConnectURLs()) {
                NatsUri nuri = new NatsUri(url);
                System.out.println(nuri + " ---> " + nuri.getUri().getScheme() + " " + nuri.getUri().getPort());
                if (nuri.hostIsIpAddress()) {
                    System.out.println("    already an ip address.");
                }
                else {
                    resolve(nuri);
                }
            }
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }

    private static void resolve(NatsUri nuri) {
        InetAddress last;
        try {
            InetAddress[] addresses = InetAddress.getAllByName(nuri.getHost());
            for (InetAddress a : addresses) {
                try {
                    NatsUri rehosted = reHost(nuri, a.getHostAddress());
                    System.out.println("    " + a.getHostAddress() + " | OK | " + rehosted);
                }
                catch (URISyntaxException u) {
                    System.out.println("    " + a.getHostAddress() + " | EX | " + u);
                }
            }
        }
        catch (UnknownHostException e) {
            System.out.println("UHE | " + e);
        }
    }

    public static NatsUri reHost(NatsUri nuri, String newHost) throws URISyntaxException {
        URI uri = nuri.getUri();
        int at = newHost.indexOf("://");  // 10.111.7.11://10.111.7.11:4222
        if (at != -1) {
            newHost = newHost.substring(0, at);
        }
        String newUrl = (uri.getRawUserInfo() == null)
            ? uri.getScheme() + "://" + newHost + ":" + uri.getPort()
            : uri.getScheme() + "://" + uri.getRawUserInfo() + "@" + newHost + ":" + uri.getPort();
        return new NatsUri(newUrl, uri.getScheme());
    }
}

[z0mb1ek]

Logs:

[scottf]

I'm going to make some assumptions, please correct me if I'm wrong:

1. what is being resolved is nats://localhost:4222
2. Given InetAddress a, a.getHostAddress() is returning something like 10.111.7.11://10.111.7.11:4222
3. So when rehost, I'm getting nats://10.111.7.11://10.111.7.11:4222:4222

I updated the test code and have a fix. It makes the assumption that if the getHostAddress has the :// that I can just ignore that part.

[scottf]

@z0mb1ek Apologies I was out of office for the last 2 weeks for a personal issue. If you can confirm the last reply item 2, I can make a fix.

[z0mb1ek]

@scottf sorry, i did not see your reply. It starts working. I can ask my devops what has changed in dns resolving