search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
29.51k stars 9.45k forks source link

Load datasets from huggingface hub #11378

Closed ryan-minato closed 1 month ago

ryan-minato commented 9 months ago

Add a loader to the mmdet.datasets module to download and load datasets from the huggingface hub.

Motivation In our use case, this will help leverage datasets to manage dataset caching, which is faster than downloading from the original server. It will also help in achieving streaming (e.g., using the WebDatasets module in datasets), which is crucial for large datasets like ImageNet.

Related resources https://huggingface.co/docs/datasets/object_detection#object-detection

Additional context I am willing to create a PR to implement this feature, although it may take some time.

ryan-minato commented 9 months ago

I have created a PR to implement this feature. It is here #11415 .