search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
471 stars 81 forks source link

api key 10241 is invalid? #109

Open gdubya opened 1 year ago

gdubya commented 1 year ago

Hi, we're trying to set up a proxy with TLS certificate authentication:

kafka-proxy server --bootstrap-server-mapping "brokers.myorg.com:9093,0.0.0.0:32399" \ --tls-enable

We use kafkacat to test the connection:

kcat -C -b $BROKER_LIST \ -t $MYTOPIC \ -s avro \ -r https://schema-registry.myorg.com \ -X security.protocol=SSL \ -X ssl.key.location=client_key.key \ -X ssl.key.password=$CLIENT_PASS\ -X ssl.certificate.location=client_cert.cer \ -X ssl.ca.location=cert_chain.pem

This works fine when BROKER_LIST=brokers.myorg.com:9093 But when I change to BROKER_LIST= localhost:32399 then the proxy throws an error message:

INFO[2022-09-06T11:08:34+02:00] New connection for brokers.myorg.com:9093 DEBU[2022-09-06T11:08:34+02:00] Kafka request key 10241, version 1, length 369295617 INFO[2022-09-06T11:08:34+02:00] Reading data from local connection on 127.0.0.1:32399 from 127.0.0.1:44186 (brokers.myorg.com:9093) had error: api key 10241 is invalid <repeat previous 2 lines>

And on the client side an error message about SSL handshake failure:

%3|1662455314.896|FAIL|rdkafka#consumer-1| [thrd:ssl://localhost:32399/bootstrap]: ssl://localhost:32399/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 126ms in state SSL_HANDSHAKE) %3|1662455314.928|FAIL|rdkafka#consumer-1| [thrd:ssl://localhost:32399/bootstrap]: ssl://localhost:32399/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 31ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed) ^C% ERROR: Failed to query metadata for topic $MYTOPIC: Local: Broker transport failure %3|1662455319.770|FAIL|rdkafka#consumer-1| [thrd:ssl://localhost:32399/bootstrap]: ssl://localhost:32399/bootstrap: No further error information available (after 0ms in state SSL_HANDSHAKE)

I should also add that the hostname "brokers.myorg.com" resolves to multiple (6) IP addresses. Does that affect how we should configure the proxy?

everesio commented 1 year ago

Option "--tls-enable" configures the proxy to use TLS when connecting to the broker. In your case the proxy listens on the "plain" port, but the kcat uses TLS when connecting to it. To configure proxy to listen on TLS use "--proxy-listener-tls-enable" and configure proxy certificates.