Azure: Support vended credentials refresh in ADLSFileIO. - Githubissues

apache / iceberg

Apache Iceberg

https://iceberg.apache.org/

Apache License 2.0

6.49k stars 2.24k forks source link

Azure: Support vended credentials refresh in ADLSFileIO. #11577

Open ChaladiMohanVamsi opened 3 days ago

ChaladiMohanVamsi commented 3 days ago

Proposed Change

Add support to refresh and consume vended storage credentials for ADLSFileIO.

New Azure properties

client.refresh-credentials-enabled property to control credential refresh and defaults to true.
client.refresh-credentials-endpoint defines endpoint to fetch the refresh credentials with below spec.
- https://github.com/apache/iceberg/pull/11281

Requirements of credential config

Required to have Sas token expiration time as part of credential config along with Sas token.
Expected prefix for the expiration time is adls.sas-token-expire-at-ms. similar to existing sas token prefix adls.sas-token.
Expected to have only one ADLS credential per storage-account prefix.

Similar PRs for other FileIOs

ChaladiMohanVamsi commented 3 days ago

cc// @nastra @jackye1995 @amogh-jahagirdar @munendrasn Can you please help with the review.