search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 86 forks source link

Running proxy with tls-enable works, but not with proxy-listener-tls-enable #83

Open enjoyear opened 3 years ago

enjoyear commented 3 years ago

Hi, I am trying to use the kafka-proxy as a way to do cross-VPC access for AWS MSK. First of all, I am trying to set up the proxy within the same VPC as the MSK. I am able to make it work with --tls-enable only. The proxy will be started as below

./kafka-proxy server \
        --bootstrap-server-mapping "b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3001" \
        --bootstrap-server-mapping "b-2.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3002" \
        --dynamic-listeners-disable \
        --kafka-client-id client \
        --tls-enable \
        --tls-ca-chain-cert-file "/home/ec2-user/ssl2/truststore-certs.pem" \
        --tls-client-cert-file "/home/ec2-user/ssl2/keystore-certs.pem" \
        --tls-client-key-file  "/home/ec2-user/ssl2/key.pem" \
        --tls-client-key-password pass123

then connect locally using bin/kafka-console-consumer.sh --bootstrap-server ip-10-1-0-94.ec2.internal:3001 --topic AWSKafkaTutorialTopic2 --from-beginning. Everything works fine.

However, when I tried to add the TLS between client and proxy, things start to break. I have done 3 experiments, and I connect to all 3 using bin/kafka-console-consumer.sh --bootstrap-server ip-10-1-0-94.ec2.internal:3001 --topic AWSKafkaTutorialTopic2 --from-beginning --consumer.config client.properties, where the client.properties works fine when I connect directly to the AWS MSK bin/kafka-console-consumer.sh --bootstrap-server b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094 --topic AWSKafkaTutorialTopic2 --from-beginning --consumer.config client.properties

Experiment 1: using port 9094

./kafka-proxy server \
        --bootstrap-server-mapping "b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3001" \
        --bootstrap-server-mapping "b-2.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3002" \
        --dynamic-listeners-disable \
        --kafka-client-id proxy \
        --proxy-listener-tls-enable \
        --proxy-listener-ca-chain-cert-file "/home/ec2-user/kafka-proxy/ssl/truststore-certs.pem" \
        --proxy-listener-cert-file "/home/ec2-user/kafka-proxy/ssl/proxyServer-keystore-certs.pem" \
        --proxy-listener-key-file  "/home/ec2-user/kafka-proxy/ssl/key.pem" \
        --debug-enable

Results

  1. Proxy log 
    INFO[2021-06-18T01:03:12Z] New connection for b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094
INFO[2021-06-18T01:03:12Z] Reading data from b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094 had error: unexpected EOF
  2. Broker log 
    [2021-06-18 01:03:12,122] INFO [SocketServer brokerId=1] Failed authentication with /INTERNAL_IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Experiment 2: using port 9092

./kafka-proxy server \
        --bootstrap-server-mapping "b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9092,0.0.0.0:3001" \
        --bootstrap-server-mapping "b-2.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9092,0.0.0.0:3002" \
        --dynamic-listeners-disable \
        --kafka-client-id proxy \
        --proxy-listener-tls-enable \
        --proxy-listener-ca-chain-cert-file "/home/ec2-user/kafka-proxy/ssl/truststore-certs.pem" \
        --proxy-listener-cert-file "/home/ec2-user/kafka-proxy/ssl/proxyServer-keystore-certs.pem" \
        --proxy-listener-key-file  "/home/ec2-user/kafka-proxy/ssl/key.pem" \
        --debug-enable

Result:

  1. Client Log 
    [2021-06-18 01:07:01,748] ERROR [Consumer clientId=consumer-console-consumer-42640-1, groupId=console-consumer-42640] Connection to node 2147483645 (/0.0.0.0:3002) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2021-06-18 01:07:01,750] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative names present
at sun.security.ssl.Alert.createSSLException(Alert.java:131)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
  2. Proxy Log 
    INFO[2021-06-18T01:07:01Z] Reading data from local connection on 10.1.0.94:3002 from 10.1.0.94:52242 (b-2.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9092) had error: tls: received unexpected handshake message of type *tls.clientHelloMsg when waiting for *tls.certificateMsg
  3. Broker Log: None

Experiment 3: combine both tls-enable and proxy-listener-tls-enable

./kafka-proxy server \
        --bootstrap-server-mapping "b-1.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3001" \
        --bootstrap-server-mapping "b-2.chen-guo-msk.abcdef.c3.kafka.us-east-1.amazonaws.com:9094,0.0.0.0:3002" \
        --dynamic-listeners-disable \
        --kafka-client-id proxy \
        --proxy-listener-tls-enable \
        --proxy-listener-ca-chain-cert-file "/home/ec2-user/kafka-proxy/ssl/truststore-certs.pem" \
        --proxy-listener-cert-file "/home/ec2-user/kafka-proxy/ssl/proxyServer-keystore-certs.pem" \
        --proxy-listener-key-file  "/home/ec2-user/kafka-proxy/ssl/key.pem" \
        --tls-enable \
        --tls-ca-chain-cert-file "/home/ec2-user/ssl2/truststore-certs.pem" \
        --tls-client-cert-file "/home/ec2-user/ssl2/keystore-certs.pem" \
        --tls-client-key-file  "/home/ec2-user/ssl2/key.pem" \
        --tls-client-key-password pass123 \
        --debug-enable

Result: exactly the same as Experiment2

I feel like there might be something wrong with how I generate the keys/certificates for the proxy, but I cannot figure out what. When I do openssl s_client -connect ip-10-1-0-94.ec2.internal:3001, the end of the response is like below

---
SSL handshake has read 18546 bytes and written 417 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
140263774463808:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42

I checked your response in https://github.com/grepplabs/kafka-proxy/issues/40#issuecomment-644652136 and tried to disable TLS1.3, but it didn't help. It's always using TLSv1.3 as I can tell through openssl. I am not sure whether TLS1.3 is the problem. Also it seems that only tls13 is included in the binary.

[ec2-user@ip-10-1-0-94 kafka-proxy]$ strings kafka-proxy  | grep -i tls13.go
/usr/local/go/src/crypto/tls/handshake_server_tls13.go
/usr/local/go/src/crypto/tls/handshake_client_tls13.go

On the other hand, from the results in Experiment 1, we can actually tell that the TLS between client and proxy should already work. Otherwise, there shouldn't be any logs on the broker side.

Any suggestions would be great! Thank you!!

jibanes commented 3 years ago

@everesio could you suggest a workaround?

jibanes commented 3 years ago

Could it be that the cipher TLS_AES_128_GCM_SHA256 not being supported? Or alternatively that the CA root certificate and the CA intermediate certificate should be imported into the truststore?

everesio commented 3 years ago

@jibanes @enjoyear Sorry for answering so late. Could you check the 0.3.0 version ?