search

edenhill / kcat

Generic command line non-JVM Apache Kafka producer and consumer
Other
5.44k stars 483 forks source link

Kcat with SSL fails #418

Open britoaldan opened 1 year ago

britoaldan commented 1 year ago

we have a kafka cluster with latest images, enabled with tls(ssl). kafka-conosle producer and consumer works fine. whereas the kcat utility fails to get the metadata information

kafkacat -b xx.xx.xx.xx:9093 -X security.protocol=SSL -X ssl.ca.location=ca.pem -L % ERROR: Failed to acquire metadata: Local: Broker transport failure

Note: client auth is diabled and ssl.endpoint.algorithm is none version of kcat 1.5.0/1.7.0 openssl version: 1.1.1-f librdkafa version: 1.8.2 confluent kafka version: 7.2.0

Error at kcat : kafkacat -b xx.xx.xx.xx:9093 -X security.protocol=SSL -X ssl.ca.location=ca.pem -L % ERROR: Failed to acquire metadata: Local: Broker transport failure

Error trace at kafka broker `{"type":"log", "host":"test-kafka-0.default", "level":"INFO", "systemid":"kafka-98aefcdc873b4bbe80ca61a6728eb4ac", "system":"kafka", "time":"2023-01-13T04:08:07.397", "timezone":"UTC", "log":{"message":"data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-1 - org.apache.kafka.common.network.Selector - [SocketServer listenerType=ZK_BROKER, nodeId=1001] Failed authentication with /172.17.0.23 (channelId=172.17.0.21:9092-172.17.0.23:45010-2) (SSL handshake failed)"}} javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.187 UTC|null:-1|Ignore unknown or unsupported extension ( "session_ticket (35)": {

} ) javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.187 UTC|null:-1|Ignore unknown or unsupported extension ( "encrypt_then_mac (22)": {

} ) javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.188 UTC|null:-1|Ignore unknown or unsupported extension ( "client_certificate_type (21)": { 0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0090: 00 00 00 ... } ) javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consuming ClientHello handshake message ( "ClientHello": { "client version" : "TLSv1.2", "random" : "1C 5B A3 37 09 DD 1F C4 1D E1 1E DF 5C 33 71 61 E9 0A 23 D6 8C 71 24 23 55 9F D6 B4 41 E6 91 CB", "session id" : "95 19 AB 65 0A BB 37 A1 21 B4 D7 A7 EB 5F 7F 5C EB 52 38 01 F9 59 E0 61 02 0E 39 AC BA 1A DC A1", "cipher suites" : "[TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_128_CCM_SHA256(0x1304), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030), TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA9), TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA8), TLS_ECDHE_ECDSA_WITH_AES_256_CCM(0xC0AD), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(0xC02F), TLS_ECDHE_ECDSA_WITH_AES_128_CCM(0xC0AC), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256(0xC023), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(0xC027), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA(0xC00A), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA(0xC014), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA(0xC009), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA(0xC013), TLS_RSA_WITH_AES_256_GCM_SHA384(0x009D), TLS_RSA_WITH_AES_256_CCM(0xC09D), TLS_RSA_WITH_AES_128_GCM_SHA256(0x009C), TLS_RSA_WITH_AES_128_CCM(0xC09C), TLS_RSA_WITH_AES_256_CBC_SHA256(0x003D), TLS_RSA_WITH_AES_128_CBC_SHA256(0x003C), TLS_RSA_WITH_AES_256_CBC_SHA(0x0035), TLS_RSA_WITH_AES_128_CBC_SHA(0x002F), TLS_DHE_RSA_WITH_AES_256_GCM_SHA384(0x009F), TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256(0xCCAA), TLS_DHE_RSA_WITH_AES_256_CCM(0xC09F), TLS_DHE_RSA_WITH_AES_128_GCM_SHA256(0x009E), TLS_DHE_RSA_WITH_AES_128_CCM(0xC09E), TLS_DHE_RSA_WITH_AES_256_CBC_SHA256(0x006B), TLS_DHE_RSA_WITH_AES_128_CBC_SHA256(0x0067), TLS_DHE_RSA_WITH_AES_256_CBC_SHA(0x0039), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]", "compression methods" : "00", "extensions" : [ "server_name (0)": { type=host_name (0), value=test-kafka-0.test-kafka-headless.default.svc.cluster.local }, "ec_point_formats (11)": { "formats": [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2] }, "supported_groups (10)": { "versions": [x25519, secp256r1, x448, secp521r1, secp384r1] }, "session_ticket (35)": {

},
"encrypt_then_mac (22)": {

},
"extended_master_secret (23)": {
  <empty>
},
"signature_algorithms (13)": {
  "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, ed25519, ed448, rsa_pss_pss_sha256, rsa_pss_rsae_sha256, rsa_pss_pss_sha384, rsa_pss_rsae_sha384, rsa_pss_pss_sha512, rsa_pss_rsae_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, ecdsa_sha224, rsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1]
},
"supported_versions (43)": {
  "versions": [TLSv1.3, TLSv1.2]
},
"psk_key_exchange_modes (45)": {
  "ke_modes": [psk_dhe_ke]
},
"key_share (51)": {
  "client_shares": [  
    {
      "named group": x25519
      "key_exchange": {
        0000: 24 6E 8A FB D7 CB 18 F0   62 5E 54 42 B9 41 81 81  $n......b^TB.A..
        0010: 49 71 17 82 B9 16 87 68   5C C1 65 DE 2C DC B4 44  Iq.....h\.e.,..D
      }
    },
  ]
},
"client_certificate_type (21)": {
  0000: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0010: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0020: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0030: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0040: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0050: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0060: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0070: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0080: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0090: 00 00 00                                           ...
}

] } ) javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: supported_versions javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Negotiated protocol version: TLSv1.3 javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: psk_key_exchange_modes javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Handling pre_shared_key absence. javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|no server name matchers, ignore server name indication javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: server_name javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: max_fragment_length javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: status_request javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: supported_groups javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: ec_point_formats javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: signature_algorithms javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: signature_algorithms_cert javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: status_request_v2 javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: extended_master_secret javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: cookie javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: certificate_authorities javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported named group: x25519 javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: key_share javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: renegotiation_info javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|use cipher suite TLS_AES_256_GCM_SHA384 javax.net.ssl|ERROR|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|Fatal (UNEXPECTED_MESSAGE): No common named group ( "throwable" : { javax.net.ssl.SSLProtocolException: No common named group at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.KeyShareExtension$HRRKeyShareProducer.produce(Unknown Source) at java.base/sun.security.ssl.SSLExtension.produce(Unknown Source) at java.base/sun.security.ssl.SSLExtensions.produce(Unknown Source) at java.base/sun.security.ssl.ServerHello$T13HelloRetryRequestProducer.produce(Unknown Source) at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goHelloRetryRequest(Unknown Source) at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source) at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source) at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:182) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at kafka.network.Processor.poll(SocketServer.scala:1144) at kafka.network.Processor.run(SocketServer.scala:1047) at java.base/java.lang.Thread.run(Unknown Source)}

) javax.net.ssl|WARNING|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|outbound has closed, ignore outbound application data `

britoaldan commented 1 year ago

hi @edenhill
could you please have a look at this