On-Premise Hazelcast on Kubernetes - Unisocket Client Approach - java.lang.IllegalStateException: Unable to connect to any address in the config

[aaranke]

I'm using the 'https://hazelcast.com/blog/how-to-set-up-your-own-on-premises-hazelcast-on-kubernetes/' 'UniSocket' approach to deploy the hazelcast cluster on local windows 10 machine using Docker Desktop which is Kubernetes enabled. Have used the rbac, config, hazelcast & hazelcast-volume YAML from this URL (https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/kubernetes) I have a SpringBoot app running on same windows 10 machine and trying to connect the SpringBoot app to this Hazelcast cluster getting below exception.

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address in the config! The following addresses were tried:[localhost/127.0.0.1:5701, localhost/127.0.0.1:5702, localhost/127.0.0.1:5703] at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ... 30 common frames omitted Caused by: java.lang.IllegalStateException: Unable to connect to any address in the config! The following addresses were tried:[localhost/127.0.0.1:5701, localhost/127.0.0.1:5702, localhost/127.0.0.1:5703] at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToOne(ClusterListenerSupport.java:215) at com.hazelcast.client.spi.impl.ClusterListenerSupport.connectToCluster(ClusterListenerSupport.java:148) at com.hazelcast.client.spi.impl.ClientClusterServiceImpl.start(ClientClusterServiceImpl.java:183) at com.hazelcast.client.impl.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:262) at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:86) at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:71) at org.springframework.boot.autoconfigure.hazelcast.HazelcastClientConfiguration$HazelcastClientConfigFileConfiguration.hazelcastInstance(HazelcastClientConfiguration.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ... 31 common frames omitted

Attaching all the YAML and also find below the sample Hazelcast Client code..

public class HazelCastManager {

    @Autowired
    private HazelcastConfig hazelcastConfig;

    private HazelcastInstance hazelcastInstance;

    @Autowired
    public HazelCastManager(HazelcastConfig hazelcastConfig) {

        this.hazelcastConfig = hazelcastConfig;

        long startTime = System.currentTimeMillis();
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
        ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
        networkConfig.addAddress(hazelcastConfig.getServer().getAddress().split(","));

        NearCacheConfig nearCacheConfig = new NearCacheConfig();
        nearCacheConfig.setName("hazelcast");
        nearCacheConfig.setEvictionPolicy(EvictionPolicy.NONE.toString());
        clientConfig.addNearCacheConfig(nearCacheConfig);

        this.hazelcastInstance = HazelcastClient.newHazelcastClient(clientConfig);

        long endTime = System.currentTimeMillis();
        log.info("createClient END : Time Taken = " + (endTime - startTime) + " ms");
    }
}

Output of kubectl get pods,deploy,svc

NAME READY STATUS RESTARTS AGE pod/hazelcast-0 0/1 Running 1 3m50s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/hazelcast NodePort 10.96.171.247 5701:30887/TCP 3m49s service/kubernetes ClusterIP 10.96.0.1 443/TCP 32h

hazelcast-unisocket.zip

[leszko]

What are the addresses in your hazelcastConfig?

This does not look correct since Hazelcast Client tries to connect to localhost/127.0.0.1:5701, localhost/127.0.0.1:5702, localhost/127.0.0.1:5703, while your NodePort is deployed on the port 30887. Also you don't disable smart routing in any place. You need to add:

clientConfig.getNetworkConfig().setSmartRouting(false);

[aaranke]

The 'hazelcastConfig' retrieves value from aplication YAML file, that have value as below.

hazelcast:
  server:
    address: localhost:5701

Also now I have disabled the smartRouting as below..

        ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
        networkConfig.addAddress(hazelcastConfig.getServer().getAddress());
        networkConfig.setSmartRouting(false);

I have removed the file 'hazelcast-client.yaml' as the configuration is done using Java code.

[leszko]

1. To me this log [localhost/127.0.0.1:5701, localhost/127.0.0.1:5702, localhost/127.0.0.1:5703] means that your YAML config is not read.

2. Also the port does not seem right, you should use your Node port instead of 5701.

[aaranke]

Have tried using the nodeport - 30887also, however no luck!

As mentioned earlier, I have removed the hazelcast-client.yaml file, as I am using Java based configuration - ClientConfig Hope that is OK?

I have also tried using 'LoadBalancer' approach as a try, but the still not able to connect.

Here's the output with 'LoadBalancer' approach...

kubectl get pods,svc,deployment

NAME              READY   STATUS    RESTARTS   AGE
pod/hazelcast-0   0/1     Running   0          4s

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/hazelcast    LoadBalancer   10.103.82.28   localhost     5701:31060/TCP   4s
service/kubernetes   ClusterIP      10.96.0.1      <none>        443/TCP          2d4h

Exception---

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.HazelCastManager]: Constructor threw exception; nested exception is java.lang.IllegalStateException: Unable to connect to any address in the config! The following addresses were tried: [[localhost]:31060]
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:310)
        ... 28 common frames omitted
Caused by: java.lang.IllegalStateException: Unable to connect to any address in the config! The following addresses were tried: [[localhost]:31060]
        at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.connectToClusterInternal(ClientConnectionManagerImpl.java:864)
        at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.access$1500(ClientConnectionManagerImpl.java:101)
        at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$3.call(ClientConnectionManagerImpl.java:874)
        at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$3.call(ClientConnectionManagerImpl.java:870)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)

Please see below the Hazelcast YAML file

hazelcast.zip

Please see below the Java code again

    public HazelCastManager(HazelcastConfig hazelcastConfig) {

        this.hazelcastConfig = hazelcastConfig;

        long startTime = System.currentTimeMillis();
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.getGroupConfig().setName("Global").setPassword("Global-pass");
        ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
        networkConfig.addAddress(hazelcastConfig.getServer().getAddress());
        networkConfig.setSmartRouting(false);

        NearCacheConfig nearCacheConfig = new NearCacheConfig();
        nearCacheConfig.setName("Global*");
        nearCacheConfig.setEvictionPolicy(EvictionPolicy.NONE.toString());
        clientConfig.addNearCacheConfig(nearCacheConfig);

        this.hazelcastInstance = HazelcastClient.newHazelcastClient(clientConfig);

        long endTime = System.currentTimeMillis();
        log.info("createClient END : Time Taken = " + (endTime - startTime) + " ms");
    }

THE YAML from from where the'address' is read

hazelcast:
  server:
    address: localhost:31060

Also see below the dependency defined in POM.xml for Client Java Code.

<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-client</artifactId>
    <version>3.9</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>3.9</version>
    <scope>compile</scope>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-all</artifactId>
    <version>3.9</version>
</dependency>

[aaranke]

Any update/further comment?

[leszko]

I've checked the following steps are they work fine on Windows 10, Docker Dektop version 2.3.3.2 (46784).

1. Start Hazelcast cluster with Helm

$ helm install hazelcast --set service.type=LoadBalancer,service.clusterIP="" hazelcast/hazelcast

2. Check the LoadBalancer address and port (it should be localhost:5701)

$ kubectl get svc hazelcast
NAME        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hazelcast   LoadBalancer   10.106.97.234   localhost     5701:30702/TCP   8m39s

3. Use the following Java code to connect to the cluster and put some values.

    public static void main(String[] args) {
        ClientConfig clientConfig = new ClientConfig();
        clientConfig.getNetworkConfig().addAddress("127.0.0.1:5701");
        clientConfig.getNetworkConfig().setSmartRouting(false);
        HazelcastInstance hazelcastInstance = HazelcastClient.newHazelcastClient(clientConfig);

        IMap<String, String> map = hazelcastInstance.getMap("map");
        for (int i = 0; i < 10_000; i++) {
            System.out.println(String.format("putting entry (key#%d, value#%d)", i, i));
            map.put("key#" + i, "value#" + i);
        }
    }

Everything works fine. So, to investigate your issue, I recommend doing the following steps.

1. Check your Docker Desktop version
2. Try to reproduce my steps if they work for you.
3. If they work, then look for the issue in the Spring part

[aaranke]

I thought you would share the anomalies/discrepancies in my code/configuration and hence it would work !

Docker desktop version I'm using is 2.3.0.4 (46911), Kubernetes version is v1.16.5

One thing that I noticed is, in the Java Code, you created ClientConfig, however you are not using the same while creating HazelcastInstance i.e. HazelcastClient.newHazelcastClient(); Is that been done knowingly or its typo error. If its typo error, how the ClientConfig will be used as it is not referred anywhere in the code.

[leszko]

Sorry, my bad. I corrected the code above. Also, you'd better use 127.0.0.1:5701 instead of localhost:5701, since sometimes Hazelcast Connection Manager has issues while resolving DNS names.

Anyway, check out the code from my response above. It's correct now.

[leszko]

@aaranke does it work for you now?

[leszko]

Closing due to inactivity