Enable estimate info (size) on all datasets

huggingface / dataset-viewer

Lightweight web API for visualizing and exploring any dataset - computer vision, speech, text, and tabular - stored on the Hugging Face Hub

https://huggingface.co/docs/datasets-server

Apache License 2.0

639 stars 65 forks source link

Closed lhoestq closed 1 week ago

lhoestq commented 1 week ago

TODO before merging:

[x] merge https://github.com/huggingface/dataset-viewer/pull/2932
[x] test json in prod (allenai/c4 en.noblocklist: correct relative error of only 0.02%)
[x] test webdataset in prod (datasets-maintainers/small-publaynet-wds-10x: correct with relative error of only 0.07%)

severo commented 1 week ago

we won't recompute for all the datasets, right? it will thus be applied progressively, when a dataset is updated.

lhoestq commented 1 week ago

Yes correct. If needed we can manually recompute for some important datasets like C4