search

kaaveland / pyarrowfs-adlgen2

Use pyarrow with Azure Data Lake gen2
MIT License
25 stars 6 forks source link

Support OneLake onelake.dfs.fabric.microsoft.com #27

Open keshavksingh opened 1 month ago

keshavksingh commented 1 month ago

The current package has a hard validation for ADLS Gen2 https://{ACCOUNT_NAME}.dfs.core.windows.net. It would be worthwhile to add the support for Microsoft Fabric OneLake which has its account as https://{ACCOUNT_NAME}.dfs.fabric.microsoft.com

kaaveland commented 3 weeks ago

This is already supported. dfs.core.windows.net is only mentioned in the alternative constructors, the from_account_name classmethods: https://github.com/kaaveland/pyarrowfs-adlgen2/blob/3ac2c0083c97920a048f5e6732044ee943e63156/pyarrowfs_adlgen2/core.py#L300

If you use the main constructor (__init__) you can pass in anything you'd like that supports the azure.storage.filedatalake.DataLakeServiceClient API: https://github.com/kaaveland/pyarrowfs-adlgen2/blob/3ac2c0083c97920a048f5e6732044ee943e63156/pyarrowfs_adlgen2/core.py#L556

You could use that like this:


client = DataLakeServiceClient("https://{ACCOUNT_NAME}.dfs.fabric.microsoft.com", azure.identity.DefaultAzureCredential())
fs = pyarrowfs_adlgen2.AccountHandler(client)