Hazelcast with Linkerd Service Mesh not working

Hi, I have some feedback about this page

I am trying to follow the tutorial and got errors when deploying hazelcast cluster. With Linkerd proxy it logs:

hazelcast-cluster 2022-07-05 10:30:05,430 [ INFO] [hz.pensive_northcutt.IO.thread-in-1] [c.h.i.s.t.TcpServerConnection]: [10.0.0.29]:5701 [dev] [4.2] Connection[id=892, /10.0.0.29:43557->/10.0.0.9:5701, qualifier=null, endpoint=[10.0.0.9]:5701, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
hazelcast-cluster 2022-07-05 10:30:05,433 [ INFO] [hz.pensive_northcutt.IO.thread-in-1] [c.h.i.s.t.TcpServerConnection]: [10.0.0.29]:5701 [dev] [4.2] Connection[id=886, /10.0.0.29:46577->/10.0.0.11:5701, qualifier=null, endpoint=[10.0.0.11]:5701, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
hazelcast-cluster 2022-07-05 10:30:05,435 [ INFO] [hz.pensive_northcutt.IO.thread-in-1] [c.h.i.s.t.TcpServerConnection]: [10.0.0.29]:5701 [dev] [4.2] Connection[id=888, /10.0.0.29:37359->/10.0.1.11:5701, qualifier=null, endpoint=[10.0.1.11]:5701, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
hazelcast-cluster 2022-07-05 10:30:05,436 [ INFO] [hz.pensive_northcutt.IO.thread-in-1] [c.h.i.s.t.TcpServerConnection]: [10.0.0.29]:5701 [dev] [4.2] Connection[id=897, /10.0.0.29:35179->/10.0.1.8:5701, qualifier=null, endpoint=[10.0.1.8]:5701, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
hazelcast-cluster 2022-07-05 10:30:05,442 [ INFO] [hz.pensive_northcutt.IO.thread-in-2] [c.h.i.s.t.TcpServerConnection]: [10.0.0.29]:5701 [dev] [4.2] Connection[id=885, /10.0.0.29:37623->/10.0.1.13:5701, qualifier=null, endpoint=[10.0.1.13]:5701, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
hazelcast-cluster 2022-07-05 10:30:05,443 [ INFO] [hz.pensive_northcutt.IO.thread-in-2] [c.h.i.s.t.TcpServerConnection]:

I cannot connect to the cluster and after some time it is stuck in crashloopbackoff.

Linkerd check output:

Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks can be verified
√ cluster networks contains all node podCIDRs

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
‼ control plane is up-to-date
    is running version 2.10.2 but the latest stable version is 2.11.2
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running stable-2.10.2 but cli running stable-2.11.2
    see https://linkerd.io/2.11/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
    * linkerd-controller-7cc5b95657-krrtc (stable-2.10.2)
    * linkerd-destination-69ff88f465-22tsg (stable-2.10.2)
    * linkerd-identity-7dd8cbcc66-7qm45 (stable-2.10.2)
    * linkerd-proxy-injector-857dfdb8cf-blsgx (stable-2.10.2)
    * linkerd-sp-validator-7d8db89555-55sk6 (stable-2.10.2)
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-controller-7cc5b95657-krrtc running stable-2.10.2 but cli running stable-2.11.2
    see https://linkerd.io/2.11/checks/#l5d-cp-proxy-cli-version for hints

Status check results are √

Other services with Linkerd work correctly. Setting linkerd proxy settings to:

config.linkerd.io/skip-outbound-ports: "5701"
config.linkerd.io/skip-inbound-ports: "5701"
config.linkerd.io/opaque-ports: "5701"

also did not help.

Any ideas or tips on how to make it work?

hazelcast-guides / linkerd

Hazelcast with Linkerd Service Mesh not working #3