Issues to execute tests on MSK Serveless Clusters - Topic are not being created

[EderRoger]

Hi,

I'm trying without sucess perform tests on our MSK IAM Serverless clusters using Kubernetes Openmessaging pods As per MSK documentation we created all the necessary IAM roles and permissions but when we are running the perf tests the topic creation step stuck in a loop. Look the stack below

Nov 09, 2023 3:29:29 PM com.amazonaws.http.AmazonHttpClient addProxyConfig INFO: Configuring Proxy. Proxy Host: <redacted> Proxy Port: 4438 15:29:29.408 [qtp1390301622-20] INFO AbstractLogin - **Successfully logged in**. 15:29:29.451 [qtp1390301622-20] WARN AdminClientConfig - The configuration 'debug' was supplied but isn't a known config. 15:29:29.453 [qtp1390301622-20] INFO AppInfoParser - Kafka version: 3.2.1 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka commitId: -- 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka startTimeMs: --- 15:29:29.499 [qtp1390301622-23] INFO WorkerHandler - Received create topics request for topics: { "numberOfTopics" : 1, "numberOfPartitionsPerTopic" : 10 } 15:29:39.518 [pool-2-thread-1] INFO KafkaTopicCreator - Created topics 0/1 ^C WE are stuck on this loop "Created topics 0/1 " even with the connection successfully with the cluster

our configs:

commonConfig: | bootstrap.servers=<msk_serverless_url>.amazonaws.com:9098 security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM debug=true sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::<role>" awsDebugCreds=true awsRoleSessionName="msk-perf-test"; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Our IAM policy has opened all permissions on Kafka to test

any idea what we can do to make this happen? We did try using serviceaccount but face same issue.

Thanks in advance

[EderRoger]

Hello everyone. As a test I created a IRSA role and a serviceaccount to the benchmark pods. From the pod I did the download for Kafka binaries and execute the topic creation and did work. ./kafka-topics.sh --bootstrap-server :9098 --create --topic test-topic --partitions 1 --replication-factor 3 --command-config client.properties

But using the openmessging framerowk did not. same looping and not creating the topics.

Any ideas?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 5 days if no further activity occurs. Thank you for your contributions.