Open kumarappan-arumugam opened 4 months ago
Looks like the auth callback is not used with the AdminClient. Not sure if there's any workaround at the moment besides using username/password auth.
Looks like this affects methods like Consumer.list_topics()
as well, it doesn't work with OAUTHBEARER
. That particular issue can be worked around by executing consumer.poll() first, after which list_topics() seems to be working fine. I've switched over to kafka-python (https://kafka-python.readthedocs.io/en/master/) for now, because this seems to be working fine there.
Facing the same issue as well when trying to connect to MSK from a lambda. However, java based lambda works just fine. I have triple checked the IAM side - both java and python lambdas are using the same SG and role. This not just happens for admin but for producer as well. Are there any fix in progress for this?
I was able to get this to work in my code by calling e.g. admin_client.poll(1)
just after instantiating the client. (In my case, I had to do it with my producer
as well.) But I think the auth mechanism might work asynchronously, because if I try to use the admin client too soon after issuing that poll()
it will fail... so for now I have a 3-second sleep after the poll()
call and so far, so good. Not great (and not documented as far as I've found), but maybe it will get you going.
I am having the same issue while trying to access MSK from an EKS node with IAM.
@woodlee I tried using your suggestion with admin_client.poll(1)
but unfortunately it did not work for me. Have you found a different way to work around this issue?
Facing the same issue. Has anyone found a work around for this ?
@ketan-nabera Actually, the workaround above also worked for me after some trial and error. In my case, I had to put a 10-second sleep after poll(1)
.
@ketan-nabera Our code that is currently working properly against an AWS MSK cluster just does an initial call to poll(3)
right before calling other admin client methods, and doesn't seem to need any additional sleep time after that. You can see it here (in the method at L473 if the link doesn't take you there).
I suppose it's possible that the needed amount of poll timeout could depend on network latency between the process and the brokers. But that's just a guess on my part. In our case the code is running on EC2 in the same region as our MSK clusters.
Description
Admin client cannot connect to the bootstrap servers when using
SASL_SSL
withOAUTHBEARER
. I'm trying to connect to AWS MSK cluster bootstrap servers. Admin client can connect when using an unauthenticated endpoint. Producer and Consumer config work fine withOAUTHBEARER
.How to reproduce
Debug logs:
Checklist
Please provide the following information:
confluent_kafka.version()
andconfluent_kafka.libversion()
): 2.3.0{...}
'debug': '..'
as necessary)