MirroMaker client authentication issue with external Kafka cluster

eazhilan-nagarajan commented 4 years ago

Hi,

I'm trying to mirror data between two Kafka cluster running in two different OpenShift clusters. I'm trying the "scram-sha-512" client authentication type but I get the below error in MirrorMaker logs.

To explain better how I did:

Source cluster:

Copied the cluster-ca-cert secret from target cluster and created a secret in the namespace where source cluster runs.
Copied the Kafka user's secret alone that contains the password from target cluster and created a secret with the same password in the namespace where source cluster runs.

MirrorMaker 2 config yaml:

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: my-mirrormaker-1
spec:
  version: 2.4.0
  replicas: 3
  resources:
    requests:
      cpu: "500m"
      memory: 2Gi
    limits:
      cpu: "1.5"
      memory: 2Gi
  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  connectCluster: "target-kafka-cluster"
  clusters:
  - alias: "source-kafka-cluster"
    authentication:
      passwordSecret:
        password: password
        secretName: mirrormaker-user-1
      type: scram-sha-512
      username: mirrormaker-user-1
    bootstrapServers: my-cluster-1-kafka-bootstrap-prod-ABC.com:443
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: my-cluster-1-cluster-ca-cert
    config:    
      consumer.exclude.internal.topics: "false"  
  - alias: "target-kafka-cluster"
    authentication:
      passwordSecret:
        password: password
        secretName: target-mirrormaker-password-secret
      type: scram-sha-512
      username: mirrormaker-user-1
    bootstrapServers: my-cluster-1-kafka-bootstrap-prod-XYZ.com:443
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: target-my-cluster-1-cluster-ca-cert
    config:
      producer.max.request.size: 15728640
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3
  mirrors:
  - sourceCluster: "source-kafka-cluster"
    targetCluster: "target-kafka-cluster"
    sourceConnector:
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 3
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    checkpointConnector:
      config:
        tasks.max: 1
        checkpoints.topic.replication.factor: 3
      tasksMax: 1
      #pause: false  
    topics.blacklist: ""
    groups.blacklist: ""
    topicsPattern: ".*"
    groupsPattern: ".*"
    exclude.internal.topics: "false"

Kafka Cluster config yaml: ( just a snippet of the config )

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster-1
spec:
  kafka:
    version: 2.4.0
    replicas: 3
    listeners:
      external:
        type: route
        authentication:
          type: scram-sha-512
      plain: {}
      tls: 
        authentication:
          type: scram-sha-512
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5

Error from logs:

2020-06-30 09:40:52,297 WARN [AdminClient clientId=adminclient-1] Unexpected error from my-cluster-1-kafka-bootstrap-prod-XYZ.com/10.xx.xx.xx; closing connection (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclient-1]
java.lang.IllegalArgumentException: Empty key
    at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)
    at org.apache.kafka.common.security.scram.internals.ScramFormatter.hi(ScramFormatter.java:76)
    at org.apache.kafka.common.security.scram.internals.ScramSaslClient.handleServerFirstMessage(ScramSaslClient.java:192)
    at org.apache.kafka.common.security.scram.internals.ScramSaslClient.evaluateChallenge(ScramSaslClient.java:133)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:474)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:474)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:381)
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:263)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:547)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
    at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
    at java.lang.Thread.run(Thread.java:748)
2020-06-30 09:40:52,298 WARN [AdminClient clientId=adminclient-1] Connection to node -1 (my-cluster-1-kafka-bootstrap-prod-XYZ.com/10.xx.xx.xx:443) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

Kindly help me with what am I doing wrong.

Thanks, Eazhilan

scholzj commented 4 years ago

This looks like something is missing for the SCRAM auth. Are you sure you configured the secrets corectly? That the names are right and they contain the right password under the right key. Maybe you can check the configuration files in /tmp/ in the MirrorMaker2 container and share them - that might show what is wrong.

eazhilan-nagarajan commented 4 years ago

Unfortunately the POD has crashed and there is no running container to pull the configuration files.

To explain a little better, below is the snippet of target alias section from MirrorMaker config file:

 - alias: "target-kafka-cluster"
    authentication:
      passwordSecret:
        password: password
        secretName: target-mirrormaker-password-secret
      type: scram-sha-512
      username: mirrormaker-user-1
    bootstrapServers: my-cluster-1-kafka-bootstrap-prod-XYZ.com:443
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: target-my-cluster-1-cluster-ca-cert
    config:
      producer.max.request.size: 15728640
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3

The target-mirrormaker-password-secret was created using oc create secret generic target-mirrormaker-password-secret --from-file=<my-password.txt>.
The my-password.txt file had the password of the kafkaUser from the target cluster
username: mirrormaker-user-1 is not created in OpenShift. Should the KafkaUser be created in the source cluster? Or how this user and the secret with password can be linked?

Thanks.

eazhilan-nagarajan commented 4 years ago

Hi @scholzj, I was able to sneak into the container and found the strimzi-mirrormaker properties file.

Found the below two line but definitely the password to connect the target cluster is missing.

source-kafka-cluster.sasl.password=kNxxxxxxxz
target-kafka-cluster.sasl.password=

scholzj commented 4 years ago

So, can you get the password secrets with oc get secret xxx -o yaml? To see the structure?

PS: For the debuging of failing container ... since you seem to be already using oc, oc debug is helpful in these cases ... it lests start a debu gversion of a crashlooping pod where you can try to run the commend, debug it, check the files etc.

eazhilan-nagarajan commented 4 years ago

You got the issue right @scholzj, Adding in some details thinking it might help somebody some day :smile:.

I tried to follow the document from Strimzi echo -n '1f2d1e2e67df' > password.txt oc create secret generic test-secret --from-file=password.txt

Upon extracting it as an yaml, found the structure as below:

apiVersion: v1
data:
  password.txt: MWYyZDFlMmU2N2Rm
kind: Secret
metadata:
  creationTimestamp: "2020-06-30T13:13:01Z"
  name: gen-secret
  namespace: aaa-bbb
type: Opaque

Note: Under data, the key is the name of the file itself "password.txt" which is not what I expected. So copied the above structure, replaced "password.txt ( filename )" to "password ( the expected key name )".

After this, the MirrorMaker was able to authenticate and it worked. My mistake it was a old document I followed :grin:

Thanks again for the quick help.

scholzj commented 4 years ago

Right. Ok. The docs says that ... but it then also says that the KAfkaConnect YAML should be:

    passwordSecret:
      secretName: _<my-secret>_
      password: _<my-password.txt>_

and not just password: password.

Anway, glad you solved it!

strimzi / strimzi-kafka-operator

MirroMaker client authentication issue with external Kafka cluster #3260