Support for MS Fabric OneLake

fsspec / adlfs

fsspec-compatible Azure Datake and Azure Blob Storage access

BSD 3-Clause "New" or "Revised" License

178 stars 104 forks source link

Support for MS Fabric OneLake #416

Open mikaelene opened 1 year ago

mikaelene commented 1 year ago

Microsoft recently announced Fabric Online which is based on Azure Data Lake Storage Gen2. It should be quite compatible with ADLS Gen2 APIs and SDKs to be compatible with existing ADLS Gen2 applications. But it looks like it has a new protocol (abfss?).

It would be great to have support for OneLake and maybe some docs on how to upload files etc.

chriklev commented 6 months ago

Is this supported now?

jkafrouni commented 5 months ago

A hacky workaround I've found is:

from adlfs import AzureBlobFileSystem

fs= AzureBlobFileSystem(account_name="onelake", anon=False)
fs.account_host = "onelake.blob.fabric.microsoft.com"
fs.do_connect()

You can then pass it for example to pandas to read abfs:// paths:

df = pd.read_parquet(path, filesystem=fs)