search

fsspec / adlfs

fsspec-compatible Azure Datake and Azure Blob Storage access
BSD 3-Clause "New" or "Revised" License
175 stars 104 forks source link

Current state of the library, milestones and current development #438

Open LuchiLucs opened 10 months ago

LuchiLucs commented 10 months ago

Please bear with me, I have never heard of fsspec and adlfs. I am trying to assess the state of this library coming frome the Python SDK for Azure Blob Storage. I used them in a personal project in order to implement concurrent (asyncio) IO of multiple blobs from Azure Cloud and now I wish to have a common interface to swap the cloud fs to a local fs, so here I am. I was wondering what is the current state, e.g.: AFAIK, this library is built on top on the Python SDK, levearing it. What does it offer though or how should it be used? Are there examples of use-cases when this wrapping is useful to download/upload multiple files concurrently or in other cases? What if the blob is no longer a blob but the user wish to optimize for particular content type? (for instance if all my blobs are .parquet files)

TomAugspurger commented 10 months ago

I think https://filesystem-spec.readthedocs.io/en/latest/ will answer most of those questions.