search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.78k stars 1.28k forks source link

Unable to produce consume with tls authentication #3765

Closed saloni-2404 closed 3 years ago

saloni-2404 commented 3 years ago

Here is my Kafka.yaml: kafkaYaml: |-

  apiVersion: kafka.strimzi.io/v1beta1
  kind: Kafka
  metadata:
    name: my-cluster1
  spec:
    kafka:
      version: 2.4.0
      replicas: 3
      listeners:
        tls: 
          authentication:
            type: tls
        #external:
        # type: loadbalancer
        # tls: true
        # authentication:
        #   type: tls
      template:
        externalBootstrapService:
          metadata:
            annotations:
              cloud.google.com/load-balancer-type: "Internal"
        perPodService:
          metadata:
            annotations:
              cloud.google.com/load-balancer-type: "Internal" 
         #plain: 
          #authentication:
            #type: scram-sha-512
         #tls:
         # authentication:
         #   type: tls
      authorization:
        type: simple   
      jmxOptions: {}
      logging:
       type: inline
       loggers:
         kafka.root.logger.level: "INFO"
      metrics: {}
      storage:
        type: persistent-claim
        size: 15Gi
        deleteClaim: false
      jvmOptions:
        gcLoggingEnabled: true
      config:
        offsets.topic.replication.factor: 1
        transaction.state.log.replication.factor: 1
        transaction.state.log.min.isr: 1
        log.message.format.version: "2.4"
        log.cleaner.delete.retention.ms: "3600000"
        log.retention.ms: "3600000"
    zookeeper:
      replicas: 3
      logging:
        type: inline
        loggers:
          zookeeper.root.logger: "DEBUG"
      storage:
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
      jmxOptions: {}        
      metrics: {}  
    entityOperator:
      topicOperator: {}
      userOperator: {}
    kafkaExporter:
      topicRegex: ".*"
      groupRegex: ".*"

My kafka-user.yaml:

kafkauserYaml: |-

  apiVersion: kafka.strimzi.io/v1beta1
  kind: KafkaUser
  metadata:
    name: my-user5
    labels:
      strimzi.io/cluster: my-cluster1
  spec:
    authentication:
      type: tls
    authorization:
      type: simple
      acls:
        - resource:
            type: topic
            name: '*'
          operation: Read
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Describe
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Write
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Create
          type: allow
        - resource:
            type: group
            name: '*'
            patternType: literal
          operation: Write
          host: "*"

Other user: kafkauserYaml: |-

  apiVersion: kafka.strimzi.io/v1beta1
  kind: KafkaUser
  metadata:
    name: my-user4
    labels:
      strimzi.io/cluster: my-cluster1
  spec:
    authentication:
      type: tls
    authorization:
      type: simple
      acls:
        - resource:
            type: topic
            name: '*'
          operation: Read
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Describe
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Write
          type: allow
        - resource:
            type: topic
            name: '*'
          operation: Create
          type: allow
        - resource:
            type: group
            name: '*'
            patternType: literal
          operation: Read
          host: "*"

Using my-user5 for producer, my-user6 for consumer,

Have run following commands after copying certs:

./bin/kafka-console-producer.sh --broker-list my-cluster1-kafka-bootstrap:9093 --topic test.aot.test \
--producer-property security.protocol=SSL \
--producer-property ssl.truststore.type=PKCS12 \
--producer-property ssl.keystore.type=PKCS12 \
--producer-property ssl.truststore.password=1234567 \
--producer-property ssl.keystore.password=1234567 \
--producer-property ssl.truststore.location=/tmp/client.truststore.p12 \
--producer-property ssl.keystore.location=/tmp/producer.keystore.p12
./bin/kafka-console-consumer.sh --bootstrap-server my-cluster1-kafka-bootstrap:9093 --topic test.aot.test \
--consumer-property security.protocol=SSL \
--consumer-property ssl.truststore.type=PKCS12 \
--consumer-property ssl.keystore.type=PKCS12 \
--consumer-property ssl.truststore.password=1234567 \
--consumer-property ssl.keystore.password=1234567 \
--consumer-property ssl.truststore.location=/tmp/client.truststore.p12 \
--consumer-property ssl.keystore.location=/tmp/consumer.keystore.p12 --from-beginning

Unable to produce/consume. Error:

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:432)
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /tmp/client.truststore.p12 of type PKCS12
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)
        at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:450)
        at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:421)
        ... 3 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /tmp/client.truststore.p12 of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69)
        ... 8 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /tmp/client.truststore.p12 of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:155)
        ... 11 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
        at sun.security.util.DerInputStream.getLength(DerInputStream.java:599)
        at sun.security.util.DerValue.init(DerValue.java:391)
        at sun.security.util.DerValue.<init>(DerValue.java:332)
        at sun.security.util.DerValue.<init>(DerValue.java:345)
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1938)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 12 more
scholzj commented 3 years ago

Can you describe how did you created the keystore and truststore files?

saloni-2404 commented 3 years ago

Pasting them as used:

kubectl get secrets -n kafka-saloni-test my-cluster1-cluster-ca-cert -o jsonpath='{.data.ca.crt}' | base64 -id > ca.crt

keytool \ -import \ -file ca.crt \ -keystore client.truststore.p12 \ -alias ca \ -storepass 1234567 \ -noprompt

kubectl cp client.truststore.p12 -n kafka-saloni-test my-cluster1-kafka-0:/tmp

kubectl get secrets -n kafka-saloni-test my-user5 -o jsonpath='{.data.user.crt}' | base64 -id > user.crt

kubectl get secrets -n kafka-saloni-test my-user5 -o jsonpath='{.data.user.key}' | base64 -id > user.key

openssl pkcs12 -export -in user.crt -inkey user.key -name user.p12 -password pass:1234567 -out producer.p12

keytool -importkeystore -alias user.p12 -deststorepass 1234567 -destkeystore producer.keystore.p12 -srcstorepass 1234567 \ -srckeystore producer.p12 -srcstoretype PKCS12 -deststoretype PKCS12

kubectl cp producer.keystore.p12 -n kafka-saloni-test my-cluster1-kafka-0:/tmp

kubectl get secrets -n kafka-saloni-test my-user4 -o jsonpath='{.data.user.crt}' | base64 -id > user.crt

kubectl get secrets -n kafka-saloni-test my-user4 -o jsonpath='{.data.user.key}' | base64 -id > user.key

openssl pkcs12 -export -in user.crt -inkey user.key -name user.p12 -password pass:1234567 -out consumer.p12

keytool -importkeystore -alias user.p12 -deststorepass 1234567 -destkeystore consumer.keystore.p12 -srcstorepass 1234567 \ -srckeystore consumer.p12 -srcstoretype PKCS12 -deststoretype PKCS12

kubectl cp consumer.keystore.p12 -n kafka-saloni-test my-cluster1-kafka-0:/tmp

scholzj commented 3 years ago

The error from the application basically says it cannot read the truststore. The command looks fine on the first look. But I wonder if you need to ass there the type as well ... something like -storetype PKCS12. MAybe you should give it a try, I do not think the keytool will detect the storetype just from the file extension so it will probably have JKS format..

saloni-2404 commented 3 years ago

ok let me try this and let you know how that goes keytool -importkeystore -alias user.p12 -deststorepass 1234567 -destkeystore producer.keystore.p12 -srcstorepass 1234567 \ -srckeystore producer.p12 -srcstoretype PKCS12 -deststoretype PKCS12 -storetype PKCS12

scholzj commented 3 years ago

I meant it mainly for the first command since that is where it was failing right now. For the keystores, the p12 files produced by openssl should work fine.

saloni-2404 commented 3 years ago

This helped! Thanks a lot.. keytool \ -import \ -file ca.crt \ -keystore client.truststore.p12 \ -alias ca \ -storepass 1234567 \ -noprompt \ -storetype PKCS12

scholzj commented 3 years ago

Great, glad it helped. Can we close this if it works now?

saloni-2404 commented 3 years ago

Yes sure! Thanks :) :)