search

grepplabs / kafka-proxy

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.
Apache License 2.0
501 stars 86 forks source link

AWS Invalid API Key. What did I miss ? #166

Open srikanthvpai opened 7 months ago

srikanthvpai commented 7 months ago

AWS MSK Provisioned cluster. One of the bootstrap servers : b-1.<>.kafka.us-east-1.amazonaws.com:9098

Started kafka proxy on an ec2 which has a role with admin control on MSK and the profile dev mentioned in this command has admin access on everything. Still it throws invalid api key. What did I miss ? Started kafka-proxy with below cluster

kafka-proxy server --bootstrap-server-mapping "b-1.<>.kafka.us-east-1.amazonaws.com:9098,<>:30001" --tls-enable --tls-insecure-skip-verify --sasl-enable --sasl-method "AWS_MSK_IAM" --sasl-aws-region us-east-1 --log-level debug --sasl-aws-profile dev

A java client tried to send 100 messages to <>:30001 using the following properties

properties.put("security.protocol","SASL_SSL"); properties.put("sasl.mechanism","AWS_MSK_IAM"); properties.put("sasl.jaas.config","software.amazon.msk.auth.iam.IAMLoginModule required;"); properties.put("sasl.client.callback.handler.class","software.amazon.msk.auth.iam.IAMClientCallbackHandler");

But the console in the kafka-proxy ec2 says :

INFO[2024-04-30T04:23:36Z] New connection for b-1.<>.kafka.us-east-1.amazonaws.com:9098 DEBU[2024-04-30T04:23:36Z] Successful IAM SASL handshake. Available mechanisms: [OAUTHBEARER AWS_MSK_IAM] DEBU[2024-04-30T04:23:36Z] Kafka request key -16639, version 1, length 369296129 INFO[2024-04-30T04:23:36Z] Reading data from local connection on <>:30001 from <>:63893 (b-1.<>.kafka.us-east-1.amazonaws.com:9098) had error: api key -16639 is invalid

I am just trying to place a proxy between the client and msk cluster. Please help.