search

microsoft / kafka-connect-cosmosdb

Kafka Connect connectors for Azure Cosmos DB
MIT License
49 stars 54 forks source link

Add Support for Service Principal Authentication in Azure Cosmos DB Source/Sink Connector #570

Closed liesai closed 4 months ago

liesai commented 4 months ago

Description: Summary This pull request proposes the addition of support for Azure Active Directory (AAD) Service Principal authentication in the Azure Cosmos DB Source Connector. This enhancement will allow users to authenticate using Service Principals, providing a more secure and manageable authentication method compared to using primary keys.

Background Currently, the Azure Cosmos DB Source Connector only supports authentication via primary keys. While this method works, it presents several security and management challenges:

Security Risks: Primary keys provide full access to Cosmos DB accounts, posing a significant risk if they are exposed. Management Overhead: Rotating keys and managing access control for multiple services and users can be cumbersome.

Service Principal authentication offers a more secure and scalable solution:

Improved Security: Service Principals can have more granular permissions, reducing the risk of unauthorized access. Simplified Management: Access can be managed via Azure AD, allowing for easier permission updates and key rotation. Implementation Details

To implement this feature, the following changes are proposed:

Configuration Options: Introduce new configuration options to provide Service Principal credentials:

cosmosdb.tenant.id: The tenant ID of the Azure AD. cosmosdb.client.id: The client ID of the Service Principal. cosmosdb.client.secret: The client secret of the Service Principal.

Authentication Logic: Modify the connector's authentication logic to support Service Principal authentication using the Azure Identity SDK.

Documentation: Update the connector documentation to include instructions for configuring Service Principal authentication.

kushagraThapar commented 4 months ago

@liesai - this feature has been developed however, on the v2 version of this kafka connector, and we plan to continue this development on the v2 version only. You can learn more about the new version here - https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-kafka-connect - this new version has all the features listed here in the changelog- https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-kafka-connect/CHANGELOG.md

Can you please give it a try to check if the v2 connector suits your needs? It is very optimized and more performant compared to this version.

kushagraThapar commented 4 months ago

You can read more about it here - https://devblogs.microsoft.com/cosmosdb/enhancements-in-the-kafka-connector-for-azure-cosmos-db-a-new-era-of-scalability-and-flexibility/

liesai commented 4 months ago

Hi. @kushagraThapar, I will close this FR knowing that it is already been worked on. This is suiting my needs, do you know if confluent will have soon access to this V2, and have you others v2s for others azure services connectors in your scope ?

kushagraThapar commented 4 months ago

@liesai - yes, we are in discussion with Confluent team to make this the default (custom and fully managed) connector in Confluent. This will take some time around 6 months I think, meanwhile, you can access the connector through maven. https://mvnrepository.com/artifact/com.azure.cosmos.kafka/azure-cosmos-kafka-connect

liesai commented 4 months ago

Thank you !