Add threshold to modalities from filetypes - Githubissues

huggingface / dataset-viewer

Lightweight web API for visualizing and exploring any dataset - computer vision, speech, text, and tabular - stored on the Hugging Face Hub

https://huggingface.co/docs/datasets-server

Apache License 2.0

642 stars 65 forks source link

Add threshold to modalities from filetypes #2969

Closed lhoestq closed 5 days ago

lhoestq commented 5 days ago

Fix modalities false positives for

I added two thresholds to get less false positives:

one general threshold to ignore file types that are <10% of the files
one additional threshold specific to images in presence of csv/json/parquet, since images are often used in README as figures

cc @severo

This should take care of most false positives, and we can refine later if needed