[Question] Not able to filter topic name with mirrorMaker2

[stchau4work]

The issue I am facing is when I specify a very specific topic ("dev") with the mirrorMaker2, there is an infinite loop of heartbeat. Therefore, I would like to ask if anyone could be able to tell me what I am missing so that I can achieve a bidirectional replication please

For instance, I was deploying Strimzi 0.17 in 2 DCs with yaml files as below:

In DC1

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: my-mm2-cluster
spec:
  version: 2.4.0
  replicas: 1
  connectCluster: "my-cluster-target"
  clusters:
  - alias: "my-cluster-source"
    bootstrapServers: 168.63.214.254:9094
  - alias: "my-cluster-target"
    bootstrapServers: 13.76.153.86:9094
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
  mirrors:
  - sourceCluster: "my-cluster-source"
    targetCluster: "my-cluster-target"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    groupsPattern: ".*"
    topicsPattern: "dev"

In DC2

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: my-mm2-cluster
spec:
  version: 2.4.0
  replicas: 1
  connectCluster: "my-cluster-target"
  clusters:
  - alias: "my-cluster-source"
    bootstrapServers: 13.76.153.86:9094
  - alias: "my-cluster-target"
    bootstrapServers: 168.63.214.254:9094
    config:
      config.storage.replication.factor: 1
      offset.storage.replication.factor: 1
      status.storage.replication.factor: 1
  mirrors:
  - sourceCluster: "my-cluster-source"
    targetCluster: "my-cluster-target"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    groupsPattern: ".*"
    topicsPattern: "dev"

[ajborley]

I think the issue you have here is due to your interchangeable use of the aliases 'my-cluster-source' and 'my-cluster-target'. MM2 appends the source alias to the topic name when mirroring it to the target, and it avoids replicating topics that already have the target alias appended in the topic name so that cycles don't occur. By naming the source cluster 'my-cluster-source' in both CRs, the heartbeat topic only ever gets the alias 'my-cluster-source' appended when it is mirrored, and the MM2 cycle check is never triggered.

My suggestion would be to give each cluster a specific alias, and use the same aliases in both CRs. This will make the MM2 cycle check work correctly and will also make it clear where each topic originally came from. For example, in DC1:

...
spec:
  connectCluster: "DC1-cluster"
  clusters:
  - alias: "DC2-cluster"
    bootstrapServers: 168.63.214.254:9094
  - alias: "DC1-cluster"
    bootstrapServers: 13.76.153.86:9094
...
  mirrors:
  - sourceCluster: "DC2-cluster"
    targetCluster: "DC1-cluster"
...

And in DC2:

...
spec:
  connectCluster: "DC2-cluster"
  clusters:
  - alias: "DC2-cluster"
    bootstrapServers: 168.63.214.254:9094
  - alias: "DC1-cluster"
    bootstrapServers: 13.76.153.86:9094
...
  mirrors:
  - sourceCluster: "DC1-cluster"
    targetCluster: "DC2-cluster"
...

The above CRs should produce mirrored topics named 'DC2-cluster.dev' and DC2-cluster.heartbeat' in the DC1 cluster, and mirrored topics named 'DC1-cluster.dev' and DC1-cluster.heartbeat' in the DC2 cluster

[stchau4work]

@ajborley Thanks for the insightful response. The heartbeat issue is resolved.

However, I found out another issue that the order in alias does matter

so in your DC2 example, it should be

...
spec:
  connectCluster: "DC2-cluster"
  clusters:
  - alias: "DC1-cluster"
    bootstrapServers: 13.76.153.86:9094
  - alias: "DC2-cluster"
    bootstrapServers: 168.63.214.254:9094
...
  mirrors:
  - sourceCluster: "DC1-cluster"
    targetCluster: "DC2-cluster"
...

otherwise, a strange error related to "mirrormaker2-cluster-offset" cant not be create / found will occur