search

Lightning-AI / litdata

Transform datasets at scale. Optimize datasets for fast AI model training.
Apache License 2.0
375 stars 43 forks source link

Using fsspec to download files #181

Open samsja opened 5 months ago

samsja commented 5 months ago

🚀 Feature

I was wondering if you planned on using fsspec for downloading file. Since fsspec is already compatible with many cloud provider it would make lit-data streaming compatible with even more cloud.

Motivation

Right now there is one downloader class for each provider, fsspec would allow to abstract it away, reducing the amount of cloud provider specific code and allowing supporting more out of the box

PS: love the work on lit data

github-actions[bot] commented 5 months ago

Hi! thanks for your contribution!, great first issue!

tchaton commented 5 months ago

Hey @samsja, feel free to contribute fsspec support

samsja commented 5 months ago

Hey @samsja, feel free to contribute fsspec support

Sure, I can give it a try

deependujha commented 3 months ago

Hey @samsja, there has been no update from your side regarding any progress.

So, I'd like to work on this.

If you have any issue, plz let me know.

samsja commented 3 months ago

Hey @samsja, there has been no update from your side regarding any progress.

So, I'd like to work on this.

If you have any issue, plz let me know.

Unfortunately, I don't have time right now to check this. Feel free to take it