Mirrormaker2 fails in two different Kubernetes clusters

cloudcafetech commented 4 years ago

I am running two different Kafka clusters (prod-cluster & sync-cluster) in two different Kubernetes Clusters. Both are identical apart from name and endpoints. Only added monitoring in prod-cluster. And Trying to Mirrormaker2 from prod-cluster to sync-cluster.

Only generated ca.crt from sync-cluster and copied to prod-cluster and finally created secret (sync-cluster-cluster-ca-cert) in prod-cluster using command (kubectl create secret generic sync-cluster-cluster-ca-cert --from-file=ca.crt)

Below detail from prod-cluster

[root@k3s kafka]# more kafka-mirror-maker-2-tls.yaml
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: mirror-maker-2
spec:
  version: 2.5.0
  replicas: 3
  connectCluster: "sync-cluster"
  clusters:
  - alias: "prod-cluster"
    bootstrapServers: prod-cluster-bootstrap-streaming.10.128.0.24.nip.io:443
    tls:
      trustedCertificates:
        - secretName: prod-cluster-cluster-ca-cert
          certificate: ca.crt
  - alias: "sync-cluster"
    bootstrapServers: sync-cluster-bootstrap-streaming.10.128.0.60.nip.io:443
    tls:
      trustedCertificates:
        - secretName: sync-cluster-cluster-ca-cert
          certificate: ca.crt
    config:
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3
  mirrors:
  - sourceCluster: "prod-cluster"
    targetCluster: "sync-cluster"
    sourceConnector:
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 3
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
    topicsPattern: ".*"
    groupsPattern: ".*"
[root@k3s kafka]#
[root@k3s kafka]#
[root@k3s kafka]# more kube-kafka.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: prod-cluster
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls:
        authentication:
          type: tls
      external:
        type: ingress
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: prod-cluster-bootstrap-streaming.10.128.0.24.nip.io
          brokers:
          - broker: 0
            host: prod-cluster-broker-0-streaming.10.128.0.24.nip.io
          - broker: 1
            host: prod-cluster-broker-1-streaming.10.128.0.24.nip.io
          - broker: 2
            host: prod-cluster-broker-2-streaming.10.128.0.24.nip.io
    authorization:
      type: simple
      superUsers:
        - CN=super-user
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        class: kubenfs-storage
        size: 500Mi
        deleteClaim: false
    metrics:
      # Inspired by config from Kafka 2.0.0 example rules:
      # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
      lowercaseOutputName: true
      rules:
      # Special cases and very specific rules
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          topic: "$4"
          partition: "$5"
      - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
        name: kafka_server_$1_$2
        type: GAUGE
        labels:
          clientId: "$3"
          broker: "$4:$5"
      - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
        name: kafka_server_$1_connections_tls_info
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
          protocol: "$4"
          cipher: "$5"
      - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>c
onnections
        name: kafka_server_$1_connections_software
        type: GAUGE
        labels:
          clientSoftwareName: "$2"
          clientSoftwareVersion: "$3"
          listener: "$4"
          networkProcessor: "$5"
      - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
        name: kafka_server_$1_$4
        type: GAUGE
        labels:
          listener: "$2"
          networkProcessor: "$3"
      # Some percent metrics use MeanRate attribute
      # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      # Generic gauges for percents
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
      - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
        name: kafka_$1_$2_$3_percent
        type: GAUGE
        labels:
          "$4": "$5"
      # Generic per-second counters with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
        name: kafka_$1_$2_$3_total
        type: COUNTER
      # Generic gauges with 0-2 key/value pairs
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
        name: kafka_$1_$2_$3
        type: GAUGE
      # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
      # Note that these are missing the '_sum' metric!
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
          "$6": "$7"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          "$6": "$7"
          quantile: "0.$8"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
        labels:
          "$4": "$5"
      - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          "$4": "$5"
          quantile: "0.$6"
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
        name: kafka_$1_$2_$3_count
        type: COUNTER
      - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
        name: kafka_$1_$2_$3
        type: GAUGE
        labels:
          quantile: "0.$4"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      class: kubenfs-storage
      size: 500Mi
      deleteClaim: false
    metrics:
      # Inspired by Zookeeper rules
      # https://github.com/prometheus/jmx_exporter/blob/master/example_configs/zookeeper.yaml
      lowercaseOutputName: true
      rules:
      # replicated Zookeeper
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
        name: "zookeeper_$2"
        type: GAUGE
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
        name: "zookeeper_$3"
        type: GAUGE
        labels:
          replicaId: "$2"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
        name: "zookeeper_$4"
        type: COUNTER
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
        name: "zookeeper_$4"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
      - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w
+)"
        name: "zookeeper_$4_$5"
        type: GAUGE
        labels:
          replicaId: "$2"
          memberType: "$3"
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

Kubernetes rosource

[root@k3s kafka]# oc get po,svc,ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME                                                READY   STATUS             RESTARTS   AGE
pod/kafka-acl-viewer-7f5b7d95bd-xgxx5               1/1     Running            0          41m
pod/mirror-maker-2-mirrormaker2-78b5d78d79-44xm2    0/1     CrashLoopBackOff   8          20m
pod/mirror-maker-2-mirrormaker2-78b5d78d79-s4qg4    0/1     CrashLoopBackOff   8          20m
pod/mirror-maker-2-mirrormaker2-78b5d78d79-vsgpm    0/1     CrashLoopBackOff   8          20m
pod/prod-bridge-bridge-6c8c7dd678-n642d             1/1     Running            1          9h
pod/prod-cluster-entity-operator-756bdcfc78-zs7zj   3/3     Running            5          9h
pod/prod-cluster-kafka-0                            2/2     Running            4          9h
pod/prod-cluster-kafka-1                            2/2     Running            4          9h
pod/prod-cluster-kafka-2                            2/2     Running            4          9h
pod/prod-cluster-kafka-exporter-59bd56d9bc-vbcdv    1/1     Running            4          9h
pod/prod-cluster-zookeeper-0                        1/1     Running            2          9h
pod/prod-cluster-zookeeper-1                        1/1     Running            1          9h
pod/prod-cluster-zookeeper-2                        1/1     Running            2          9h
pod/strimzi-cluster-operator-68b6d59f74-tn69l       1/1     Running            1          9h

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
service/kafka-acl-viewer                        NodePort    10.96.249.79    <none>        80:32492/TCP                          41m
service/mirror-maker-2-mirrormaker2-api         ClusterIP   10.96.60.132    <none>        8083/TCP                              20m
service/prod-bridge-bridge-service              ClusterIP   10.96.42.195    <none>        8080/TCP                              9h
service/prod-cluster-kafka-0                    ClusterIP   10.96.152.243   <none>        9094/TCP                              9h
service/prod-cluster-kafka-1                    ClusterIP   10.96.39.161    <none>        9094/TCP                              9h
service/prod-cluster-kafka-2                    ClusterIP   10.96.154.195   <none>        9094/TCP                              9h
service/prod-cluster-kafka-bootstrap            ClusterIP   10.96.186.142   <none>        9091/TCP,9092/TCP,9093/TCP,9404/TCP   9h
service/prod-cluster-kafka-brokers              ClusterIP   None            <none>        9091/TCP,9092/TCP,9093/TCP            9h
service/prod-cluster-kafka-exporter             ClusterIP   10.96.143.233   <none>        9404/TCP                              9h
service/prod-cluster-kafka-external-bootstrap   ClusterIP   10.96.90.196    <none>        9094/TCP                              9h
service/prod-cluster-zookeeper-client           ClusterIP   10.96.187.101   <none>        9404/TCP,2181/TCP                     9h
service/prod-cluster-zookeeper-nodes            ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP            9h

NAME                                              CLASS    HOSTS                                                 ADDRESS     PORTS     AGE
ingress.extensions/kafka-acl-viewer               <none>   kafkaview-streaming.35.239.239.46.nip.io              localhost   80        41m
ingress.extensions/prod-bridge-ingress            <none>   prod-bridge-streaming.10.128.0.24.nip.io              localhost   80        9h
ingress.extensions/prod-cluster-kafka-0           <none>   prod-cluster-broker-0-streaming.10.128.0.24.nip.io    localhost   80, 443   9h
ingress.extensions/prod-cluster-kafka-1           <none>   prod-cluster-broker-1-streaming.10.128.0.24.nip.io    localhost   80, 443   9h
ingress.extensions/prod-cluster-kafka-2           <none>   prod-cluster-broker-2-streaming.10.128.0.24.nip.io    localhost   80, 443   9h
ingress.extensions/prod-cluster-kafka-bootstrap   <none>   prod-cluster-bootstrap-streaming.10.128.0.24.nip.io   localhost   80, 443   9h

Error logs

[root@k3s kafka]# oc logs -f mirror-maker-2-mirrormaker2-78b5d78d79-44xm2
Preparing MirrorMaker 2.0 cluster truststores and keystores
Preparing MirrorMaker 2.0 truststores and keystores for cluster prod-cluster
  with trusted certs prod-cluster-cluster-ca-cert/ca.crt
  with tls auth certs
  with tls auth keys
Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Preparing MirrorMaker 2.0 truststores and keystores for cluster sync-cluster
  with trusted certs sync-cluster-cluster-ca-cert/ca.crt
  with tls auth certs
  with tls auth keys
Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Preparing MirrorMaker 2.0 cluster truststores is complete
Creating connector configuration:
# TLS / SSL
ssl.truststore.password=[hidden]

Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=sync-cluster-bootstrap-streaming.10.128.0.60.nip.io:443
# REST Listeners
rest.port=8083
rest.advertised.host.name=10.244.0.24
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
config.storage.topic=mirrormaker2-cluster-configs
group.id=mirrormaker2-cluster
status.storage.topic=mirrormaker2-cluster-status
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
offset.storage.topic=mirrormaker2-cluster-offsets
config.providers=file
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
admin.security.protocol=SSL
# TLS / SSL
ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
ssl.truststore.password=[hidden]
ssl.truststore.type=PKCS12

producer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
producer.ssl.truststore.password=[hidden]

consumer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
consumer.ssl.truststore.password=[hidden]

admin.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
admin.ssl.truststore.password=[hidden]

2020-10-03 01:24:19,297 INFO WorkerInfo values:
        jvm.args = -Xms128M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
        jvm.spec = Oracle Corporation, OpenJDK 64-Bit Server VM, 11.0.7, 11.0.7+10-LTS
        jvm.classpath = /opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/bcpkix-jdk15on-1.62.jar:/opt/kafka/bin/../libs/bcprov-jdk15on-1.60.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-2.5.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.5.0.jar:/opt/kafka/bin/../libs/connect-file-2.5.0.jar:/opt/kafka/bin/../libs/connect-json-2.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-2.5.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.5.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.5.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.5.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.0.108.jar:/opt/kafka/bin/../libs/gson-2.8.6.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.12-2.10.2.jar:/opt/kafka/bin/../libs/jaeger-client-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-core-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.1.0.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.1.0.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.28.jar:/opt/kafka/bin/../libs/jersey-common-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/kafka/bin/../libs/jersey-server-2.28.jar:/opt/kafka/bin/../libs/jetty-client-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-http-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-io-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-security-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-server-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jetty-util-9.4.24.v20191120.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.12.0.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/json-smart-1.1.1.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent.jar:/opt/kafka/bin/../libs/kafka-clients-2.5.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.5.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.12-2.5.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.5.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.5.0.jar:/opt/kafka/bin/../libs/kafka_2.12-2.5.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.12-2.5.0.jar:/opt/kafka/bin/../libs/keycloak-common-10.0.0.jar:/opt/kafka/bin/../libs/keycloak-core-10.0.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.3.50.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.3.50.jar:/opt/kafka/bin/../libs/libthrift-0.13.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/opt/kafka/bin/../libs/okhttp-4.2.2.jar:/opt/kafka/bin/../libs/okio-2.2.2.jar:/opt/kafka/bin/../libs/opa-authorizer-0.4.1.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.12.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.12-2.1.3.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/opt/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.12.10.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/kafka/bin/../libs/tracing-agent.jar:/opt/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/kafka/bin/../libs/zookeeper-3.5.7.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.4-7.jar
        os.spec = Linux, amd64, 3.10.0-1127.19.1.el7.x86_64
        os.vcpus = 1
 (org.apache.kafka.connect.runtime.WorkerInfo) [main]
2020-10-03 01:24:19,302 INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2020-10-03 01:24:24,468 INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@3d4eac69 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,468 INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,469 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,470 INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,471 INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,473 INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,474 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,475 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,478 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,479 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2020-10-03 01:24:24,550 INFO DistributedConfig values:
        access.control.allow.methods =
        access.control.allow.origin =
        admin.listeners = null
        bootstrap.servers = [sync-cluster-bootstrap-streaming.10.128.0.60.nip.io:443]
        client.dns.lookup = default
        client.id =
        config.providers = [file]
        config.storage.replication.factor = 3
        config.storage.topic = mirrormaker2-cluster-configs
        connect.protocol = sessioned
        connections.max.idle.ms = 540000
        connector.client.config.override.policy = None
        group.id = mirrormaker2-cluster
        header.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
        heartbeat.interval.ms = 3000
        inter.worker.key.generation.algorithm = HmacSHA256
        inter.worker.key.size = null
        inter.worker.key.ttl.ms = 3600000
        inter.worker.signature.algorithm = HmacSHA256
        inter.worker.verification.algorithms = [HmacSHA256]
        internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
        internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
        key.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
        listeners = null
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        offset.flush.interval.ms = 60000
        offset.flush.timeout.ms = 5000
        offset.storage.partitions = 25
        offset.storage.replication.factor = 3
        offset.storage.topic = mirrormaker2-cluster-offsets
        plugin.path = [/opt/kafka/plugins]
        rebalance.timeout.ms = 60000
        receive.buffer.bytes = 32768
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 40000
        rest.advertised.host.name = 10.244.0.24
        rest.advertised.listener = null
        rest.advertised.port = 8083
        rest.extension.classes = []
        rest.host.name = null
        rest.port = 8083
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        scheduled.rebalance.max.delay.ms = 300000
        security.protocol = SSL
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.client.auth = none
        ssl.enabled.protocols = [TLSv1.2]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.2
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = PKCS12
        status.storage.partitions = 5
        status.storage.replication.factor = 3
        status.storage.topic = mirrormaker2-cluster-status
        task.shutdown.graceful.timeout.ms = 5000
        topic.tracking.allow.reset = true
        topic.tracking.enable = true
        value.converter = class org.apache.kafka.connect.converters.ByteArrayConverter
        worker.sync.timeout.ms = 3000
        worker.unsync.backoff.ms = 300000
 (org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2020-10-03 01:24:24,554 INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [main]
2020-10-03 01:24:24,558 INFO AdminClientConfig values:
        bootstrap.servers = [sync-cluster-bootstrap-streaming.10.128.0.60.nip.io:443]
        client.dns.lookup = default
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = SSL
        security.providers = null
        send.buffer.bytes = 131072
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.2
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = PKCS12
 (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,073 WARN The configuration 'producer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,073 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,073 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,073 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,073 WARN The configuration 'config.providers' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'admin.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'consumer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'producer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'config.providers.file.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'admin.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'consumer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,075 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'header.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'ssl.truststore.type' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,081 WARN The configuration 'admin.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2020-10-03 01:24:25,082 INFO Kafka version: 2.5.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-03 01:24:25,082 INFO Kafka commitId: 66563e712b0b9f84 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-03 01:24:25,082 INFO Kafka startTimeMs: 1601688265081 (org.apache.kafka.common.utils.AppInfoParser) [main]
2020-10-03 01:24:25,714 INFO [AdminClient clientId=adminclient-1] Failed authentication with sync-cluster-bootstrap-streaming.10.128.0.60.nip.io/10.128.0.60 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclient-1]
2020-10-03 01:24:25,718 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (sync-cluster-bootstrap-streaming.10.128.0.60.nip.io/10.128.0.60:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2020-10-03 01:24:25,719 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:260)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:425)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:363)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:286)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:174)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1272)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)
        at java.base/java.lang.Thread.run(Thread.java:834)
2020-10-03 01:24:25,725 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1601688295087) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
2020-10-03 01:24:25,725 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1601688295723) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
2020-10-03 01:24:25,731 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
        ... 3 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:260)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:438)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:425)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:509)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:363)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:286)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:174)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1272)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)
        at java.base/java.lang.Thread.run(Thread.java:834)

Below detail from sync-cluster

[root@kind2 kafka]# more kube-kafka.yaml
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: sync-cluster
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls:
        authentication:
          type: tls
      external:
        type: ingress
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: sync-cluster-bootstrap-streaming.10.128.0.60.nip.io
          brokers:
          - broker: 0
            host: sync-cluster-broker-0-streaming.10.128.0.60.nip.io
          - broker: 1
            host: sync-cluster-broker-1-streaming.10.128.0.60.nip.io
          - broker: 2
            host: sync-cluster-broker-2-streaming.10.128.0.60.nip.io
    authorization:
      type: simple
      superUsers:
        - CN=super-user
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        class: kubenfs-storage
        size: 500Mi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      class: kubenfs-storage
      size: 500Mi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Kubernetes resources

[root@kind2 kafka]# oc get po,svc,ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME                                                READY   STATUS    RESTARTS   AGE
pod/kafka-acl-viewer-f79b7fff6-4gxjv                1/1     Running   0          51m
pod/strimzi-cluster-operator-68b6d59f74-ppp4f       1/1     Running   0          53m
pod/sync-cluster-entity-operator-59b97b8c88-c9bss   3/3     Running   0          51m
pod/sync-cluster-kafka-0                            2/2     Running   0          51m
pod/sync-cluster-kafka-1                            2/2     Running   0          51m
pod/sync-cluster-kafka-2                            2/2     Running   0          51m
pod/sync-cluster-zookeeper-0                        1/1     Running   0          52m
pod/sync-cluster-zookeeper-1                        1/1     Running   0          52m
pod/sync-cluster-zookeeper-2                        1/1     Running   0          52m

NAME                                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/kafka-acl-viewer                        NodePort    10.96.164.146   <none>        80:31901/TCP                 51m
service/sync-cluster-kafka-0                    ClusterIP   10.96.14.42     <none>        9094/TCP                     51m
service/sync-cluster-kafka-1                    ClusterIP   10.96.73.105    <none>        9094/TCP                     51m
service/sync-cluster-kafka-2                    ClusterIP   10.96.144.108   <none>        9094/TCP                     51m
service/sync-cluster-kafka-bootstrap            ClusterIP   10.96.52.130    <none>        9091/TCP,9092/TCP,9093/TCP   51m
service/sync-cluster-kafka-brokers              ClusterIP   None            <none>        9091/TCP,9092/TCP,9093/TCP   51m
service/sync-cluster-kafka-external-bootstrap   ClusterIP   10.96.173.203   <none>        9094/TCP                     51m
service/sync-cluster-zookeeper-client           ClusterIP   10.96.200.110   <none>        2181/TCP                     52m
service/sync-cluster-zookeeper-nodes            ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   52m

NAME                                              CLASS    HOSTS                                                 ADDRESS     PORTS     AGE
ingress.extensions/kafka-acl-viewer               <none>   kafkaview-streaming.34.122.50.159.nip.io              localhost   80        51m
ingress.extensions/sync-cluster-kafka-0           <none>   sync-cluster-broker-0-streaming.10.128.0.60.nip.io    localhost   80, 443   51m
ingress.extensions/sync-cluster-kafka-1           <none>   sync-cluster-broker-1-streaming.10.128.0.60.nip.io    localhost   80, 443   51m
ingress.extensions/sync-cluster-kafka-2           <none>   sync-cluster-broker-2-streaming.10.128.0.60.nip.io    localhost   80, 443   51m
ingress.extensions/sync-cluster-kafka-bootstrap   <none>   sync-cluster-bootstrap-streaming.10.128.0.60.nip.io   localhost   80, 443   51m

cloudcafetech commented 4 years ago

Able to fix that issue.

But after replication noticed that topic name changed in target (sync-cluster)

Source Cluster (prod-cluster)

[root@prod-cluster kafka]# oc get k
NAME           DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS
prod-cluster   3                        3

[root@prod-cluster kafka]# oc get ku
NAME                      AUTHENTICATION   AUTHORIZATION
kafka-acl-viewer          tls              simple
order-confirmation-mail   tls              simple
prod-bridge               tls              simple
shipment-api              tls              simple
super-user                tls
webshop                   tls              simple

[root@prod-cluster kafka]# oc get kt
NAME                                     PARTITIONS   REPLICATION FACTOR
mm2-offset-syncs.sync-cluster.internal   1            3
sales                                    10           3
shipments                                10           2
time-tracking                            1            1
users                                    10           3

Target Cluster (sync-cluster)

[root@sync-cluster kafka]# oc get k
NAME           DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS
sync-cluster   3                        3

[root@sync-cluster kafka]# oc get ku
NAME               AUTHENTICATION   AUTHORIZATION
kafka-acl-viewer   tls              simple
super-user         tls

[root@sync-cluster kafka]# oc get kt
NAME                                                          PARTITIONS   REPLICATION FACTOR
consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a   50           3
heartbeats                                                    1            3
mirrormaker2-cluster-configs                                  1            3
mirrormaker2-cluster-offsets                                  25           3
mirrormaker2-cluster-status                                   5            3
prod-cluster.checkpoints.internal                             1            3
prod-cluster.sales                                            10           3
prod-cluster.shipments                                        10           3
prod-cluster.time-tracking                                    1            3
prod-cluster.users                                            10           3

one more observation Data not replicating ..

Few Question from my end ...

Why topic name gt changed?
How can we keep same topic name?
How can I replicate my users and ACLs?
How can I make my target cluster identical as source cluster (looking for disaster recovery (DR) scenario)?

scholzj commented 4 years ago

MM2 is always renaming topics. Since Strimzi 0.20.0 you would be able to use the IdenttiyReplication policy to keep the names the same (see how it will looks like int he CR: https://github.com/strimzi/strimzi-kafka-operator/blob/0600fd441a5652718a40b303485a9710f62994d7/examples/mirror-maker/kafka-mirror-maker-2-custom-replication-policy.yaml#L27). But that is not supported in 0.19.

To replicate users / ACLs you would need to copy the KafkaUser. Or alternatively you would need something like OAuth authentication and Keycloak authorization with once central Keycloak server.

You should also be careful about the topics since MM2 will not replicate the replication factor but use the default in the cluster instead. You can see how the replication factor changes in your topics in the outputs you copy pasted above.

cloudcafetech commented 4 years ago

Thanks for reply,

once topic replication done, I put some data in source topic and after long wait also I observed no data replication in target topic. so I want to point out data not replicating.

As you mentioned its going to introduce in the IdenttiyReplication policy to keep the names in 0.20.0, I expect that not going to mandatory features. Because this (renaming topic) helps us to prevent topic corruption (due to duplicate) when we will do bidirectional MM2.

scholzj commented 4 years ago

As you mentioned its going to introduce in the IdenttiyReplication policy to keep the names in 0.20.0, I expect that not going to mandatory features. Because this (renaming topic) helps us to prevent topic corruption (due to duplicate) when we will do bidirectional MM2.

Yes, that would be optional ... as I pointed to the example YAML.

cloudcafetech commented 4 years ago

Any update on below ..

once topic replication done, I put some data in source topic and after long wait also I observed no data replication in target topic. so I want to point out data not replicating.

ajborley commented 4 years ago

This may be a manifestation of a Kafka bug that is described here: https://github.com/strimzi/strimzi-kafka-operator/issues/3688

Does data begin replicating properly if you reduce the spec.replicas field to 1? (You may need to wait a few minutes for it to start after reducing the replicas)

cloudcafetech commented 4 years ago

you mean to say replicas (3 to 1)? ... will try

can I add refresh.topics.internval.seconds: 30 in spec.config field ? and test ... not sure does it support ?

ajborley commented 4 years ago

To change the refresh.topics.interval.seconds setting , add it to your KafkaMirrorMaker2 CR spec.mirrors.sourceConnector.config e.g.:

...
  mirrors:
  - sourceCluster: "source"
    sourceConnector:
      config:
        refresh.topics.interval.seconds: 60

cloudcafetech commented 4 years ago

Tested, working ...

Thanks Andwer 👍

strimzi / strimzi-kafka-operator

Mirrormaker2 fails in two different Kubernetes clusters #3742