search

Lightning-AI / torchmetrics

Machine learning metrics for distributed, scalable PyTorch applications.
https://lightning.ai/docs/torchmetrics/
Apache License 2.0
2.15k stars 409 forks source link

CLIPImageQualityAssessment insists on prompt argument to be called 'beutiful' #2243

Closed spezialspezial closed 1 year ago

spezialspezial commented 1 year ago

🐛 Bug

not working: metric = CLIPImageQualityAssessment(prompts=("beautiful",))

working: metric = CLIPImageQualityAssessment(prompts=("beutiful",))

This uses ClipVision, right? I assume it's not trained on 'beutiful' and there is just a typo in the CLIPImageQualityAssessment class

Expected behavior

prompt 'beautiful' should be valid

Environment

Linux, Python 3.10.13, torch 2.1.0+cu118, pip torchmetrics 1.2.0

SkafteNicki commented 1 year ago

Hi @spezialspezial, thanks for reporting this issue. I sadly cannot reproduce the error. For me it works with "beautiful" and does not work with "beutiful": image Here is the specific dictionary that we check prompts against: https://github.com/Lightning-AI/torchmetrics/blob/0cb4c7924f31751e468b474189e85c4eb5dc5824/src/torchmetrics/functional/multimodal/clip_iqa.py#L42-L59 as you can see it says beautiful in the dictionary.

SkafteNicki commented 1 year ago

I rechecked and it was fixed in this PR: https://github.com/Lightning-AI/torchmetrics/pull/2103 That change is available in the newest release of torchmetrics v1.2.1 released today. Closing issue.

spezialspezial commented 12 months ago

Thanks a lot