Multiple gateway replicas may not work gossip algorithm

[npepinpe]

Description

Nodes in a Zeebe cluster use SWIM (i.e. gossip) to propagate information about other nodes. This is how topology is propagated, and how which node is subscribed to which notification event as well. Within SWIM, each node is identified by its memberId; if two nodes share the same memberId, then problems can arise as for the other nodes they are essentially the same. As you can imagine, that has some pretty bad consequences, such as breaking topology (e.g. incomplete topologies) or missing long polling notifications.

To fix this, we simply need to ensure every gateway replica gets a different memberId. This can be done by setting the env var ZEEBE_GATEWAY_CLUSTER_MEMBERID to the hostname or some such. I think at the moment it is unfortunately hardcoded to be zeebe-gateway, which could lead to the issues I've described above if someone uses more than one gateway replica.

Note that this does not affect embedded gateways.

/cc @salaboy @Zelldon

[Zelldon]

I think this is not the case, see https://github.com/camunda-community-hub/camunda-cloud-helm/blob/main/charts/zeebe-cluster-helm/templates/gateway-deployment.yaml#L50

A deployment will have an unique name example

    "metadata": {
        "creationTimestamp": "2021-11-17T14:39:30Z",
        "generateName": "medic-cw-46-9643b40976-benchmark-zeebe-gateway-548f748bd9-",
        "labels": {
            "app.kubernetes.io/component": "gateway",
            "app.kubernetes.io/instance": "medic-cw-46-9643b40976-benchmark",
            "app.kubernetes.io/managed-by": "Helm",
            "app.kubernetes.io/name": "zeebe-cluster-helm",
            "pod-template-hash": "548f748bd9"
        },
        "name": "medic-cw-46-9643b40976-benchmark-zeebe-gateway-548f748bd9-sdxdp",