search

encord-team / text-to-image-eval

Evaluate custom and HuggingFace text-to-image/zero-shot-image-classification models like CLIP, SigLIP, DFN5B, and EVA-CLIP. Metrics include Zero-shot accuracy, Linear Probe, Image retrieval, and KNN accuracy.
https://encord.com
Apache License 2.0
35 stars 1 forks source link

Add support for multi-label classification datasets like `alkzar90/NIH-Chest-X-ray-dataset` from Hugging Face #58

Open eloy-encord opened 6 months ago

eloy-encord commented 6 months ago

When adding the JSON definition of the alkzar90/NIH-Chest-X-ray-dataset Hugging Face medical dataset, found that it's a multi-label dataset instead of a multiclass.

JSON definition:

{
  "dataset_type": "HFDataset",
  "title": "NIH-Chest-X-ray",
  "title_in_source": "alkzar90/NIH-Chest-X-ray-dataset",
  "name": "image-classification",
  "target_feature": "labels",
  "trust_remote_code": true
}

Would be great to consider adding support for multi-label datasets, where this dataset may serve as the starting test dataset.