Closed Johnson-yue closed 2 months ago
Hi, I want to finetune clip model, using your code(ft-B-train-OpenAI-CLIP-ViT-L-14.py), but in this code it compute matrix multiplication twice !
first time: in clip library, already using image_embedding and text_embedding compute matrix multiplication model.forward( image, text) in your code https://github.com/zer0int/CLIP-fine-tune/blob/2df5d36e8e5b9e65ed66632e4b552e39ed8be50d/ft-B-train-OpenAI-CLIP-ViT-L-14.py#L332 and return logits_per_image and logits_per_texts.
logits_per_image
logits_per_texts
second time: in your code https://github.com/zer0int/CLIP-fine-tune/blob/2df5d36e8e5b9e65ed66632e4b552e39ed8be50d/ft-B-train-OpenAI-CLIP-ViT-L-14.py#L196-L202, using logits_per_image and logits_per_text compute matrix multiplication, second time in function:ContrastiveLoss,
ContrastiveLoss
Is this code duplicated in ContrastiveLoss function
Hi, I want to finetune clip model, using your code(ft-B-train-OpenAI-CLIP-ViT-L-14.py), but in this code it compute matrix multiplication twice !
first time: in clip library, already using image_embedding and text_embedding compute matrix multiplication model.forward( image, text) in your code https://github.com/zer0int/CLIP-fine-tune/blob/2df5d36e8e5b9e65ed66632e4b552e39ed8be50d/ft-B-train-OpenAI-CLIP-ViT-L-14.py#L332 and return
logits_per_imageandlogits_per_texts.second time: in your code https://github.com/zer0int/CLIP-fine-tune/blob/2df5d36e8e5b9e65ed66632e4b552e39ed8be50d/ft-B-train-OpenAI-CLIP-ViT-L-14.py#L196-L202, using logits_per_image and logits_per_text compute matrix multiplication, second time in function:
ContrastiveLoss,Is this code duplicated in
ContrastiveLossfunction