search

aws / aws-msk-iam-sasl-signer-python

Apache License 2.0
32 stars 10 forks source link

Cannot authenticate to Kafka cluster #34

Closed DavidConnack closed 1 month ago

DavidConnack commented 1 month ago

Description

from confluent_kafka import Consumer
import socket
from aws_msk_iam_sasl_signer import MSKAuthTokenProvider

def oauth_cb(oauth_config):
    print(oauth_config)
    auth_token, expiry_ms = MSKAuthTokenProvider.generate_auth_token_from_profile(
        aws_profile="profile_name", region="us-east-1"
    )
    print(auth_token, expiry_ms / 1000)
    return auth_token, expiry_ms / 1000

c = Consumer(
    {
        # "debug": "all",
        "bootstrap.servers": "broker.kafka.us-east-1.amazonaws.com:9098",
        "client.id": socket.gethostname(),
        "security.protocol": "SASL_SSL",
        "sasl.mechanisms": "OAUTHBEARER",
        "oauth_cb": oauth_cb,
        "group.id": "mygroup",
        "auto.offset.reset": "earliest",
    }
)

c.subscribe(["topic"])

print("Starting consumer!")

msg = c.poll(5)

if msg.error():
    print("Consumer error: {}".format(msg.error()))
print("Received message: {}".format(msg.value().decode("utf-8")))

c.close()

What I Did

When I run the above script, I get a valid base64 token, but the script fails to connect to the cluster. I have full admin permissions on the aws account.

%6|1723368048.139|FAIL|Davids-MacBook-Pro.local#consumer-1| [thrd:sasl_ssl://broker.kafka.us-east-1.amaz]: sasl_ssl://broker.kafka.us-east-1.amazonaws.com:9098/bootstrap: Disconnected (after 458ms in state AUTH_HANDSHAKE)
%3|1723368048.139|FAIL|Davids-MacBook-Pro.local#consumer-1| [thrd:sasl_ssl://broker.kafka.us-east-1.amaz]: sasl_ssl://broker.kafka.us-east-1.amazonaws.com:9098/bootstrap: SASL OAUTHBEARER mechanism handshake failed: Local: Broker transport failure: broker's supported mechanisms: (n/a) (after 0ms in state DOWN)
%6|1723368049.414|FAIL|Davids-MacBook-Pro.local#consumer-1| [thrd:sasl_ssl://broker.kafka.us-east-1.amaz]: sasl_ssl://broker.kafka.us-east-1.amazonaws.com:9098/bootstrap: Disconnected (after 458ms in state AUTH_HANDSHAKE)
%3|1723368049.414|FAIL|Davids-MacBook-Pro.local#consumer-1| [thrd:sasl_ssl://broker.kafka.us-east-1.amaz]: sasl_ssl://broker.kafka.us-east-1.amazonaws.com:9098/bootstrap: SASL OAUTHBEARER mechanism handshake failed: Local: Broker transport failure: broker's supported mechanisms: (n/a) (after 0ms in state DOWN)
Traceback (most recent call last):
  File "/Users/david/Documents/Repos/Morphisec/server2.monorepo/python/elastic-assets-job/kafka.py", line 34, in <module>
    if msg.error():
       ^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'error'
DavidConnack commented 1 month ago

After going through this with AWS support, I discovered that this was due to the fact that the Kafka version we were using was not supported, although I did not see anything in the documentation indicating this limitation. After upgrading to version 3.7.x I can connect using the iam auth

github-actions[bot] commented 1 month ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.