Unable to connect to Hazelcast Exporter in AKS

[ReeceKindon]

Hi,

We have successfully deployed Zeebe Cluster & Simple Monitor into AKS using the zeebe/zeebe-cluster-helm & zeebe-simple-monitor-helm charts, but we are struggling connecting the 2 and getting Simple Monitor to start.

The Broker has an initContainer section in the yaml to download the zeebe-hazelcast-exporter.jar and mount at "/exporters" which we can confirm has succeeded via the logs and exec'ing into the Pod's themselves. On the Broker "stateful set" we have added the below env variables:

- name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
  value: exporters/zeebe-hazelcast-exporter.jar
- name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
  value: io.zeebe.hazelcast.exporter.HazelcastExporter
- name: ZEEBE_HAZELCAST_PORT
  value: '5701'

When starting up Simple Monitor we are able to connect to the Broker fine, but unable to connect to the Hazelcast Exporter. These are our env variables for Simple Monitor deployment:

env:
  - name: io.zeebe.monitor.connectionString
    value: 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:26500'
  - name: io.zeebe.monitor.hazelcast.connection
    value: 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:5701'

This is a dump from the logs in Simple Monitor pod:

2021-02-19 15:25:16.799  INFO 1 --- [           main] i.z.m.zeebe.ZeebeConnectionService       : Connecting to broker 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:26500'
2021-02-19 15:25:17.389  INFO 1 --- [           main] i.z.m.zeebe.ZeebeConnectionService       : connected to 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:26500'
2021-02-19 15:25:17.389  INFO 1 --- [           main] i.z.m.zeebe.ZeebeConnectionService       : connected to 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:26500'
2021-02-19 15:25:17.403  INFO 1 --- [           main] i.z.m.zeebe.ZeebeConnectionService       : Connecting to Hazelcast 'zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/usr/local/zeebe-simple-monitor.jar!/BOOT-INF/lib/hazelcast-3.12.4.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-02-19 15:25:17.882  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Exception during initial connection to [zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local/10.0.72.169:5701
2021-02-19 15:25:17.882  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Unable to get live cluster connection, retry in 3000 ms, attempt 1 of 2.
2021-02-19 15:25:20.883  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Exception during initial connection to [zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local/10.0.72.169:5701
2021-02-19 15:25:20.884  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Unable to get live cluster connection, attempt 2 of 2.
2021-02-19 15:25:20.884  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Unable to connect to any address for cluster: dev. The following addresses were tried: [[zeebecluster4-zeebe-gateway.zeebe4.svc.cluster.local]:5701]
2021-02-19 15:25:20.884  WARN 1 --- [ient_0.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_0 [dev] [3.12.4] Could not connect to any cluster, shutting down the client: Unable to connect to any cluster.

We are now stuck at this point and despite various attempts to expose port 5701 on the Broker and Gateway and also different connection strings, we haven't had any luck.

Any ideas on what to try next or where this issue resides? Zeebe Broker, Hazelcast Exporter, Zeebe Gateway or Simple Monitor?

Thanks,

Reece

[saig0]

Related forum post: https://forum.camunda.io/t/can-simple-monitor-be-deployed-to-aks/1017

[nitram509]

@ReeceKindon is this issue still present, or can we close this issue?

[nitram509]

No feedback/response. Closing for now. If still relevant, please feel free to re-open.