search

LAION-AI / CLIP_benchmark

CLIP-like model evaluation
MIT License
590 stars 74 forks source link

Attribute based eval #72

Open rom1504 opened 1 year ago

rom1504 commented 1 year ago

https://arxiv.org/abs/2209.06103

JeniaJitsev commented 1 year ago

Rationale: introduce a "true" zero-shot eval into the benchmark, based on attribute-based zero-shot task.

The task is harder, as in contrast to standard zero-shot transfer used currently for CLIP eval, it requires to deal with new classes which were never presented during pre-training by describing any entity to be recognized by a set of attributes (eg sandy yellowish, furry, four legged, fast, strong) instead of using a class label (eg lion). For more details, see https://arxiv.org/abs/2209.06103

mehdidc commented 1 year ago

That would be great, is the code available somewhere ? @hildekuehne

hildekuehne commented 1 year ago

Will check and get back soon.