search

encord-team / text-to-image-eval

Evaluate custom and HuggingFace text-to-image/zero-shot-image-classification models like CLIP, SigLIP, DFN5B, and EVA-CLIP. Metrics include Zero-shot accuracy, Linear Probe, Image retrieval, and KNN accuracy.
https://encord.com
Apache License 2.0
33 stars 0 forks source link

Adding support for Hybrid-CLIP models such as MedCLIP variants also Evaluation Data saving. #83

Open TennoSerra opened 2 months ago

TennoSerra commented 2 months ago

When doing some work testing multimodal transformer models in the medical field sometimes the models in question use Hybrid-Clip variants, such as these: https://huggingface.co/models?search=medclip. It would be great if some models like these could be supported.

More importantly, is there any way to save evaluation data to a csv or json file?

eloy-encord commented 1 month ago

Hi @TennoSerra, I'm glad that TTI-Eval helps you with your workflow! To register models from known sources to TTI-Eval, you can use the following guide.

In the particular case of MedCLIP, you may first need to add a model source that knows how to interact with the details of the medclip package and then register the model instance. Contributions are always more than welcome 🤗!

On the other side, to save the evaluation results to a CSV, use the --save option of the evaluate command.