Open ShashidharC opened 2 weeks ago
Since you mention this works well without transactions, I suspect it could be because of this reason mentioned in https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html
For Kafka versions 2.8.0 and above, the WriteDataIdempotently permission is deprecated (KIP-679). By default,enable.idempotence = true is set. Therefore, for Kafka versions 2.8.0 and above, IAM does not offer the same functionality as Kafka ACLs. It is not possible to WriteDataIdempotently to a topic by only providing WriteData access to that topic. This does not affect the case when WriteData is provided to ALL topics. In that case, WriteDataIdempotently is allowed. This is due to differences in implementation of IAM logic versus how the Kafka ACLs are implemented.
To work around this, we recommend using a policy similar to the sample below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:AlterCluster",
"kafka-cluster:DescribeCluster",
"kafka-cluster:WriteDataIdempotently"
],
"Resource": [
"arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
]
},
{
"Effect": "Allow",
"Action": [
"kafka-cluster:*Topic*",
"kafka-cluster:WriteData",
"kafka-cluster:ReadData"
],
"Resource": [
"arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1/TestTopic"
]
}
]
}
In this case, WriteData allows writes to TestTopic, while WriteDataIdempotently allows idempotent writes to the cluster. It is important to note that WriteDataIdempotently is a cluster level permission. It cannot be used at the topic level. If WriteDataIdempotently is restricted to the topic level, this policy will not work.
Can you check if modifying permissions this way works for you?
Yes. @sankalpbhatia . We gave WriteDataIdempotently at cluster level. Not topic level. Also we have access to all the topics that we created.
Description
We are using MSK cluster and doing IAM auth. However, when we implement the transactional producer, it fails to connect. It errors out saying.
When I remove the transactional behaviour it works well without any issue.
Producer conf:
AWS POST: https://repost.aws/questions/QUzHXpsvHDQOiLafIUL-VgFQ/kafka-transactions-not-working-while-using-iam-authentication-with-conlfuent-kafka-python-producer-client Stack overflow: https://stackoverflow.com/questions/78021233/aws-msk-transactions-support
Could you please take a look?