Sync-Catalog Syncer.go to support update strategies upsert to service catalog

Feature Scenario

Two Kubernetes clusters that represent a hybrid environment, writing concurrently to the same consul external server. What happens is that each kubernetes cluster then invalidates the existing services from the other cluster, concurrently updating the catalog with only the services they see on their own cluster.

The request is to be able to define how Sync catalog handles merge strategy, being able to use an upsert strategy we can have services from multiple clusters without having to federate them or do a peering process both that are significantly more complex than adding a parameter to the sync catalog agent config.

What we need is a way to upsert services from the two clusters and not have the two sync catalog agents rewrite each other. Similar to how externalDNS works for DNS records.

Is there a way to define the merge or update strategy of the consul -sync catalog agent?

Or do we always have to run multiple consul servers and federate them to have a proper catalog for the same environment (hybrid)?

Helm Configuration

# Contains values that affect multiple components of the chart.
global:
  logLevel: debug
  # The main enabled/disabled setting.
  # If true, servers, clients, Consul DNS and the Consul UI will be enabled.
  enabled: true
  # The prefix used for all resources created in the Helm chart.
  name: consul
  # The consul image version.
  image: hashicorp/consul:1.15.2
  # The name of the datacenter that the agents should register as.
  datacenter: dc1
  # Enables TLS across the cluster to verify authenticity of the Consul servers and clients.
  tls:
    enabled: false
  # Enables ACLs across the cluster to secure access to data and APIs.
  acls:
    # If true, automatically manage ACL tokens and policies for all Consul components.
    manageSystemACLs: false
# Configures values that configure the Consul server cluster.
server:
  enabled: false
  # The number of server agents to run. This determines the fault tolerance of the cluster.
  #replicas: 3
externalServers:
  enabled: true
  hosts: ["1.1.1.1"]
  https: false
  httpsPort: 8500
  k8sAuthMethodHost: https://IlikeConsul:8500
# Contains values that configure the Consul UI.
#ui:
#  enabled: true
  # Registers a Kubernetes Service for the Consul UI as a LoadBalancer.
#  service:
#    type: LoadBalancer
# Configures and installs the automatic Consul Connect sidecar injector.
connectInject:
  enabled: false
# Enable catalog sync on cc-dev namespace only
syncCatalog:
  enabled: true
  default: true
  k8sAllowNamespaces: ['dev']

Current understanding and Expected behavior Each of the Sync-catalog agents only sees its services, it will invalidate the services the other agent has written and this will happens concurrently.

Expected behavior is that services are only deregistered if they are unhealthy. So the merge strategy should be upsert, similar to how externalDNS can be configured.

hashicorp / consul-k8s

Sync-Catalog Syncer.go to support update strategies upsert to service catalog #3291