search

jina-ai / clip-as-service

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
https://clip-as-service.jina.ai
Other
12.48k stars 2.07k forks source link

multiclass classification with ranking #798

Closed RaiAmanRai closed 2 years ago

RaiAmanRai commented 2 years ago

Hi @ZiniuYu , is there any way for ranking sentences given an image, with multi-class classification.

I noticed that currently it takes the sum of scores and calculates percentage probablity by dividing the value with the total sum. But any way for multi-class where the total probablity need not necessarily sum to 1.

ZiniuYu commented 2 years ago

Hi @RaiAmanRai , welcome to CLIP-as-service! Softmax scores should be able to do classifications. However, you can still use .scores['clip_score_cosine'] to get raw scores. Are there any special reasons for getting them?

RaiAmanRai commented 2 years ago

@ZiniuYu thanks for your quick reply. As you mentioned .scores['clip_score_cosine'] , I did come across them but couldn't figure out any way to make sense out of it.

So, when dealing with probablities, if in case there are multiple texts that could relate to the given picture, the overall probablities get distributed over all input texts. Now if I try to apply a threshold say 0.5 to the results, inorder to be a bit more sure about the prediction, the text with the highest probablity couldn't make the cut, because the probablity was distributed over three closely related texts say 0.3, 0.3,0.3 and 0.1 (for the rest of the classes).

And so, I am looking for a way to get around it.

ZiniuYu commented 2 years ago

You could try to use better-designed prompts/texts to improve the results. This is in fact a common classification problem. Feel free to test it on the Playground 🤩

RaiAmanRai commented 2 years ago

@ZiniuYu Thanks for helping this one out.