consul-sync multi k8s cluster unstable

kong62 commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Reproduction Steps

Logs

cluster 01:

2021-07-29T03:17:49.396Z [INFO]  to-consul/sink: registering services
2021-07-29T03:18:04.489Z [INFO]  to-consul/sink: registering services
2021-07-29T03:18:04.489Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-1b98f3842a91 service-consul-namespace=""
2021-07-29T03:18:04.496Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-a4e8454056c2 service-consul-namespace=""
2021-07-29T03:18:04.500Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-e5cce37d8007 service-consul-namespace=""
2021-07-29T03:18:19.592Z [INFO]  to-consul/sink: registering services
2021-07-29T03:18:34.664Z [INFO]  to-consul/sink: registering services
2021-07-29T03:18:34.664Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-1b98f3842a91 service-consul-namespace=""
2021-07-29T03:18:34.670Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-a4e8454056c2 service-consul-namespace=""
2021-07-29T03:18:34.675Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-e5cce37d8007 service-consul-namespace=""
2021-07-29T03:18:49.761Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:04.833Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:04.833Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-1b98f3842a91 service-consul-namespace=""
2021-07-29T03:19:04.838Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-a4e8454056c2 service-consul-namespace=""
2021-07-29T03:19:04.841Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-e5cce37d8007 service-consul-namespace=""
2021-07-29T03:19:19.924Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:35.005Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:35.005Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-1b98f3842a91 service-consul-namespace=""
2021-07-29T03:19:35.011Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-a4e8454056c2 service-consul-namespace=""
2021-07-29T03:19:35.015Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-e5cce37d8007 service-consul-namespace=""
2021-07-29T03:19:50.105Z [INFO]  to-consul/sink: registering services
2021-07-29T03:20:05.180Z [INFO]  to-consul/sink: registering services
2021-07-29T03:20:05.180Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-1b98f3842a91 service-consul-namespace=""
2021-07-29T03:20:05.190Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-a4e8454056c2 service-consul-namespace=""
2021-07-29T03:20:05.194Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster02 service-id=kubernetes-default-e5cce37d8007 service-consul-namespace=""

cluster 02:

2021-07-29T03:19:18.843Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:18.843Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-198b5a053404 service-consul-namespace=""
2021-07-29T03:19:18.853Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-5c94a96f78a1 service-consul-namespace=""
2021-07-29T03:19:18.861Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-fc3caffd7ddd service-consul-namespace=""
2021-07-29T03:19:33.897Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:33.897Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-198b5a053404 service-consul-namespace=""
2021-07-29T03:19:33.910Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-5c94a96f78a1 service-consul-namespace=""
2021-07-29T03:19:33.917Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-fc3caffd7ddd service-consul-namespace=""
2021-07-29T03:19:48.955Z [INFO]  to-consul/sink: registering services
2021-07-29T03:19:48.955Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-5c94a96f78a1 service-consul-namespace=""
2021-07-29T03:19:48.967Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-fc3caffd7ddd service-consul-namespace=""
2021-07-29T03:19:48.977Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-198b5a053404 service-consul-namespace=""
2021-07-29T03:20:04.010Z [INFO]  to-consul/sink: registering services
2021-07-29T03:20:04.011Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-5c94a96f78a1 service-consul-namespace=""
2021-07-29T03:20:04.020Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-fc3caffd7ddd service-consul-namespace=""
2021-07-29T03:20:04.027Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-198b5a053404 service-consul-namespace=""
2021-07-29T03:20:19.066Z [INFO]  to-consul/sink: registering services
2021-07-29T03:20:19.067Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-198b5a053404 service-consul-namespace=""
2021-07-29T03:20:19.077Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-5c94a96f78a1 service-consul-namespace=""
2021-07-29T03:20:19.084Z [INFO]  to-consul/sink: deregistering service: node-name=k8s-sync-cluster01 service-id=kubernetes-default-fc3caffd7ddd service-consul-namespace=""

Expected behavior

Environment details

k8s cluster 01 ----> consul-sync 1 ----
                                             |
                                              ----> consul
                                             |
k8s cluster 02 ----> consul-sync 2 ----

consul-k8s version: hashicorp/consul-k8s:0.26.0
consul-helm version: 0.32.1

cluster01： helm deploy consul and consul-sync

values.yaml used to deploy the helm chart:

server:
enabled: true
syncCatalog:
enabled: true
default: true
toConsul: true
toK8S: false
consulNodeName: "k8s-sync-cluster01"

cluster02： deploy consul-sync only by k8s deployment

  containers:
  - command:
    - /bin/sh
    - -ec
    - |
      consul-k8s sync-catalog \
        -k8s-default-sync=true \
        -to-k8s=false \
        -consul-domain=consul \
        -allow-k8s-namespace="*" \
        -deny-k8s-namespace="kube-system" \
        -deny-k8s-namespace="kube-public" \
        -k8s-write-namespace=${NAMESPACE} \
        -node-port-sync-type=ExternalFirst \
        -log-level=info \
        -consul-node-name=k8s-sync-cluster02 \
        -add-k8s-namespace-suffix \
        -consul-write-interval=15s \
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: CONSUL_HTTP_ADDR
      value: http://121.41.116.241:8500
    image: hashicorp/consul-k8s:0.26.0

Additional Context

service instances very unstable ：

this is my want：

ndhanushkodi commented 3 years ago

Hi @kong62, the workflow you describe:

k8s cluster 01 ----> consul-sync 1 ----
                                             |
                                              ----> consul
                                             |
k8s cluster 02 ----> consul-sync 2 ----

is unfortunately not supported at this time. The behaviour you are seeing is because each syncer is trying to delete the services that the other syncer has synced. Each syncer is programmed to keep a list of services it has synced, and will delete anything not synced by it.

If you would like services in multiple clusters synced to Consul, you could consider using Consul Service mesh, and using Federation between Kubernetes Clusters to have all of the services registered in Consul.

Nitya and @sadjamz

kong62 commented 3 years ago

@ndhanushkodi slaved, thanks

cluster01：

            -consul-node-name=k8s-sync-cluster01 \
            -consul-k8s-tag=cluster01 \

cluster02：

            -consul-node-name=k8s-sync-cluster02 \
            -consul-k8s-tag=cluster02 \

bondido commented 2 years ago

Hi, @ndhanushkodi - is that way - described by @kong62 - of differentiating k8s clusters by -consul-k8s-tag a supported way of syncing services from multiple k8s clusters to single consul datacenter? Can we use it and be sure it won't stop working (being "fixed" as a bug) in some future version?

If solving similar cases by appropriate tagging is not really taking advantage of a "bug" - could it be described in docs?

lkysow commented 2 years ago

Hi @bondido, yes that's the expected way. It won't be removed later. I can open up a ticket to document this better but if you'd like to submit a documentation PR yourself the page is here: https://github.com/hashicorp/consul/blob/main/website/content/docs/k8s/service-sync.mdx

hashicorp / consul-k8s