AWS MSK Serverless - had error: api key -13567 is invalid

[ayanevbg]

Hi,

I have an issue trying to communicate from a client to a MSK Serverless cluster though the kafka-proxy.

The proxy is running as a pod inside a cluster. The nodes are in the same VPC as the MSK cluster. The IAM permissions are set correctly I am able to produce on a cluster without the proxy.

Proxy version - 0.3.6

Args:                                                                                                                                                                                                                                                                    │
│       server                                                                                                                                                                                                                                                                 │
│       --bootstrap-server-mapping=<msk-serverless>.amazonaws.com:9098,0.0.0.0:32400                                                                                                                                                           │
│       --tls-insecure-skip-verify                                                                                                                                                                                                                                             │
│       --tls-enable                                                                                                                                                                                                                                                           │
│       --sasl-enable                                                                                                                                                                                                                                                          │
│       --sasl-method=AWS_MSK_IAM                                                                                                                                                                                                                                              │
│       --sasl-aws-region=ap-northeast-1                                                                                                                                                                                                                                       │
│       --log-level=debug

I am running this command from another pod to test the connection

kafka-topics.sh --create --bootstrap-server kafka-serverless-proxy:9098 --topic test-topic --command-config /etc/config/client-ssl.properties --partitions 3

The client--ssl.properties

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

I am gettig the following error in the proxy:

│ time="2023-10-23T12:31:06Z" level=info msg="Starting kafka-proxy version v0.3.6"                                                          
│ time="2023-10-23T12:31:06Z" level=info msg="Bootstrap server <msk-serverless>.amazonaws.com:9098 advertised as 0.0.0.0:32400"                                                                                                               
│ time="2023-10-23T12:31:06Z" level=info msg="Listening on 0.0.0.0:32400 (:32400) for remote <msk-serverless>.amazonaws.com:9098"                                                                                                             
│ time="2023-10-23T12:31:06Z" level=info msg="Ready for new connections"                                                                                                                                                                                               
│ time="2023-10-23T12:31:33Z" level=debug msg="Successful IAM SASL handshake. Available mechanisms: [AWS_MSK_IAM]"   
│ time="2023-10-23T12:31:33Z" level=debug msg="Kafka request key -13567, version 1, length 369296129"                                                                                                                                            
│ time="2023-10-23T12:31:33Z" level=info msg="Reading data from local connection on <pod-ip>:32400 from <client-ip>:49148 (<msk-serverless>.amazonaws.com:9098) had error: api key -13567 is invalid"                                      
│ time="2023-10-23T12:31:34Z" level=info msg="New connection for <msk-serverless>.amazonaws.com:9098"                                                                                                                                          
│ time="2023-10-23T12:31:34Z" level=debug msg="Successful IAM SASL handshake. Available mechanisms: [AWS_MSK_IAM]"

On the client side I get this warning:

WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-serverless-proxy/:9098) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)

I am not sure is this is some misconfiguration or its an issue with the serverless MSK working with the proxy. Please advice ?

Thank you!

[magiusdarrigo]

I was able to have it working with a similar setup to yours. Except I never had those client--ssl.properties on the client because that is the job of the proxy, from my understanding.

If I were you, I would first test to see if an instance running a simple kafkacat can reach your MSK cluster through the proxy.

[srikanthvpai]

Hello @ayanevbg Were you able to fix this issue ?

Regards

[ayanevbg]

Hello @ayanevbg Were you able to fix this issue ?

Regards

No, I managed to expose the Serverless MSK dns through an Inbound Resolver and with custom core dns config the dns was resolvable in clusters in other VPCs.

So, the proxy was no longer needed for our use case.

[srikanthvpai]

Will we be able to attach it to any NLB when you have inbound resolvers ? Reason I ask: how can we expose it if its through endpoint. Curious to know what was the endpoint type and protocol if you used any.