search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.62k stars 802 forks source link

object_store Azure Government using OAuth #6759

Open zadeluca opened 2 days ago

zadeluca commented 2 days ago

Which part is this question about object_store Azure

Describe your question I have reviewed this issue https://github.com/apache/arrow-rs/issues/4853 which also pertains to using Azure Government. That example uses shared key authentication. For OAuth (ClientSecretOAuthProvider), I believe it is also necessary to set authority_host to authority_hosts::AZURE_GOVERNMENT in order for the token_url to be correct.

Without this setting, I am currently getting this error "AADSTS900382: Confidential Client is not supported in Cross Cloud request. I understand it is possible to override using MicrosoftAzureBuilder::with_authority_host(), however I am using delta-rs which requires that all configuration is provided though storage_options with keys defined in AzureConfigKey. This currently does not allow setting AuthorityHost.

Does that sound correct? If so, is it possible to expose AuthorityHost through AzureConfigKey?

Additional context I have only tried in Python using delta-rs. I am going to attempt to use arrow-rs directly in order to verify that setting AuthorityHost does indeed resolve the error mentioned above. I am a Rust beginner so it may take some effort.

tustvold commented 1 day ago

I'm not very familiar with Azure's government offering, but we should expose the option as a config key if it isn't

zadeluca commented 12 hours ago

I have confirmed that including both of these:

            .with_authority_host("https://login.microsoftonline.us")
            .with_endpoint("https://<ACCOUNT>.blob.core.usgovcloudapi.net".to_string())

on the builder allows OAuth to succeed. I will try to submit a PR to add AuthorityHost to AzureConfigKey.