search

backprop-ai / backprop

Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.
https://backprop.co
Other
242 stars 12 forks source link

Finetuning Image Text Vectorizer with CLIP #21

Open singularity014 opened 2 years ago

singularity014 commented 2 years ago

Hello, I tried finetuning Image-Text Vectorizer CLIP model using above approach. But I get stuck with the error -

image

Link to full code - Colab

What I need is something which gives cosine similarity between an image and a text, shall I finetune with triplet, or with cosine similarity? if its cosine similarity, then how will I get those cosine similarity?

The triplet variant takes text and image and gives one normalised vector, I am bit confused because I thought it would give a cosine similarity.

ojasaar commented 2 years ago

Hey,

Your code is not public so I can't see it. You should be able to finetune with either. Cosine similarity is just a function between two vectors.

See an example of it being used here: https://github.com/backprop-ai/backprop/blob/main/examples/ImageVectorisation.ipynb