Connect to Event Hubs for Kafka with a SAS Token?

[archetec]

Description

Our security team wants us to use SAS tokens with a TTL to connect to event hubs. At the moment we use Confluent's .Net Kafka client library to connect to Event Hubs (we want to retain compatibility with Kafka). So we tried to connect to Event Hubs for Kafka with a SAS token but it doesn't work. Is this supported through the Kafka API?

It works with the MS Event Hubs client (i.e. without Kafka)...

Confluent Kafka 0.11.6 .Net 4.6.1 Windows 7

How to reproduce

We connect a Producer to Event Hubs for Kafka as we usually do, but using a connection string like this :

Endpoint=sb://abc12345.servicebus.windows.net/;SharedAccessSignature=SharedAccessSignature sr=https%3A%2F%2Fabc12345.servicebus.windows.net%2Feh-poc%2F&sig=5unU3lx1eSebGuFqTCi8a8R%2B0GPkSu2Xvc%2FalWHhFyA%3D&se=1550259640&skn=Manager

        var config = new Dictionary<string, object> {
            { "bootstrap.servers", brokerList },
            { "security.protocol", "SASL_SSL" },
            { "sasl.mechanism", "PLAIN" },
            { "sasl.username", "$ConnectionString" },
            { "sasl.password", connectionString },
            { "ssl.ca.location", caCertLocation },
            { "debug", "broker" }
        };

The access token was obtain from TokenProvider.CreateSharedAccessSignatureTokenProvider().

When ran, the Producer will connect partially to the broker, but fails while sending messages.

The log shows this error :

7|2019-02-15 15:06:24.075|rdkafka#producer-1|BROKERFAIL| [thrd:sasl_ssl://abc12345.servicebus.windows.net:9093/bo]: sasl_ssl://abc12345.servicebus.windows.net:9093/bootstrap: failed: err: Local: Broker handle destroyed: (errno: Invalid argument)

Has it worked previously?

No

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] Verified that port 9093 is not blocked by firewall
• [x] Verified the namespace is either Standard or Dedicated tier (i.e. it is not Basic tier, which isn't supported)
• [x] Sample you're having trouble with: <REPLACE with e.g., Java quickstart>
• [x] Apache Kafka version: <REPLACE with e.g., 1.1.0>
• [x] Kafka client configuration: <REPLACE with e.g., auto.reset.offset=earliest, ..> (do not include your connection string or SAS Key)
• [x] Namespace and EventHub/topic name
• [x] Consumer or producer failure <REPLACE with e.g., Consumer failure>
• [ ] If consumer, partition ID, group ID <REPLACE with e.g., partitionID=3, groupID=consumer-group>
• [ ] Timestamp in UTC <REPLACE wtih e.g., Nov 7 2018 - 17:15:01 UTC>
• [ ] Client ID <REPLACE with e.g., clientID=kafka-client>
• [x] Provide all logs (with debugging enabled if possible) or exception call stack
• [ ] Standalone repro <REPLACE with e.g., Willing/able to send scenario to repro issue>
• [x] Operating system: <REPLACE with e.g., Ubuntu 16.04.5 (x64) LTS>
• [ ] Critical issue

[archetec]

Can I get an official answer from the team on this please? Thank you.

[iutipikin]

@archetec Hello! My team also experienced same error when we tried to connect to EventHub using Spring Kafka client version 2.2.4. Thru logs it is looks like client cannot obtain metadata from message bus and instead get unexpected EOF with disconnect. Native EventHub client works, but it contain a lot of hardcoded stuff and force us to use Blob Storage for handling bus state, and that's we want to avoid.

[ShubhaVijayasarathy]

Adding to our user voice list.

[gholliday]

This is not supported when using the Event Hubs for Kafka endpoint.

The FAQ has been updated to reflect this:

https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-frequently-asked-questions#does-azure-event-hubs-for-apache-kafka-support-shared-access-signature-token-authentication-

Does Azure Event Hubs for Apache Kafka support shared access signature token authentication?

Authenticating by using OAuth 2.0 and shared access signature is supported.

Shared access signature tokens are generated by using an authorization rule and one of its signing keys. This is not supported when using the Event Hubs for Apache Kafka endpoint.

[nestorvw]

Is there any intention to support it?