search

LAION-AI / Big-Interleaved-Dataset

Big-Interleaved-Dataset
Apache License 2.0
58 stars 8 forks source link

Feature: Unified api for HTTP vs S3 connection #15

Open harry-stark opened 1 year ago

harry-stark commented 1 year ago

https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining

Need to use ffspec

harry-stark commented 1 year ago

https://github.com/rom1504/cc2dataset/blob/main/cc2dataset/main.py#L170