Azure Blob/ADLS Gen 2 and abfs - add instructions on using SAS token

[juwalter]

Page

https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Blob-Store.html https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Data-Lake-Gen2.html

maybe also: https://docs.alluxio.io/os/user/stable/en/ufs/Azure-Data-Lake.html

Summary

Above page describes how to configure Azure Blob and Data Lake Gen2 as "Storage Integration"; they include instructions for using "Shared Key", "OAuth 2.0 Client Credentials", and "Azure Managed Identities" for authentication. However, "SAS token" is missing on both, while - in theory - this should be possible according to:

• https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#access-azure-data-lake-storage-gen2-or-blob-storage-using-a-sas-token
• https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html

I have tried like so:

bin/alluxio fs mount \
--option fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net=SAS \
--option fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net=org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider \
--option fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net="my sas token" \
  /mnt/abfs abfs://<my container>@<storage-account>.dfs.core.windows.net/<my path>/

and also (blob.core.windows.net instead of dfs.core.windows.net and wasb instead of abfs)

bin/alluxio fs mount \
--option fs.azure.account.auth.type.<storage-account>.blob.core.windows.net=SAS \
--option fs.azure.sas.token.provider.type.<storage-account>.blob.core.windows.net=org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider \
--option fs.azure.sas.fixed.token.<storage-account>.blob.core.windows.net="my sas token" \
  /mnt/abfs wasb://<my container>@<storage-account>.blob.core.windows.net/<my path>/

blob.core.windows.net/wasb result in error message after running into a timeout:

no route to host http://169.254.169.254/metadata/... 

this looks like it is falling back to MSI (managed identity) authentication (since 169.254.169.254 only responds from inside a VM running in Azure)

dfs.core.windows.net and abfs result in a non-descript, general error

Failed get FileSystem for abfs:

I wonder if this can be done?

[Jackson-Wang-7]

I think we may not support this way right now. now we only support ClientCredsTokenProvider and MsiTokenProvider for abfs.

[juwalter]

@Jackson-Wang-7 thank you for the quick feedback

[Jackson-Wang-7]

@juwalter If you are interested, you can contribute relevant code implementations. It doesn't seem too difficult.

[juwalter]

@Jackson-Wang-7 - yes, we are looking into it!

[LuQQiu]

juwalter this is a question better suitable for alluxio.io/slack #troubleshooting channel. Greg (Email: greg.palmer@alluxio.com @gregpalmr) has more information on Azure SAS token, feel free to contact him directly.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

[iRevive]

/assign @iRevive

[juwalter]

/assign @juwalter

[juwalter]

I have created a PR for this at https://github.com/Alluxio/alluxio/pull/17583