search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.3k stars 1.96k forks source link

[FEATURE REQ] Extend AAD support to azure-cosmos-spark throughput control #41543

Closed stevenylai closed 2 weeks ago

stevenylai commented 3 weeks ago

Is your feature request related to a problem? Please describe. Due to a recent company's policy, we are no longer allowed to use master key to access CosmosDB. We are able to migrate this by using ServicePrinciple for connection but the same didn't work for our ThroughputControl and we had to disable it for now. Error message:

IllegalArgumentException: The config property 'spark.cosmos.throughputcontrol.auth.aad.clientId' is invalid. ...

Describe the solution you'd like Extend AAD control to throughput control as well. Instead of spark.cosmos.throughputControl.accountKey, we should also be able to set these:

I can see the following are required when using SP:

Perhaps we can put those under spark.cosmos.throughputControl.account (e.g. spark.cosmos.throughputControl.account.subscriptionId) ?

Describe alternatives you've considered Given that the company is disabling master key access. The only alternative is to turn off throughput control and manage it from the application

Additional context Add any other context or screenshots about the feature request here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

github-actions[bot] commented 3 weeks ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

kushagraThapar commented 3 weeks ago

@xinlian12 can you please take a look at this, thanks!

stevenylai commented 2 weeks ago

Actually one of the teammates found that if spark.cosmos.throughputControl.accountEndpoint is not set, then throughputControl will default to spark.cosmos.accountEndpoint with the same AAD-based authentication. In our application, we are not using a different account for throughput control so by omitting spark.cosmos.throughputControl.accountEndpoint it solves our problem.

With that said, I still think from a consistency perspective. If it is allowed to have throughput control on a different CosmosDB account, then the same AAD-based authentication should be enabled. But since we don't require this for now, I will close the issue.