finetuning using both itm & itc using huggingface implementation

[aretius]

Hello I would like to thank you for providing such a great paper to the community. I had a few questions

• In your paper you mention that for text to image retrieval you train on both ITC(Contrastive Loss) and ITC(Sigmoid Loss). Did you try finetuning on just 1 loss and evaluating for the same? How were the results?
• In you huggingface implementation how does one go about training with the strategy you adopted here?

[LooperXX]

Hello, Thank you for your interests in our work

• Here we follow ALBEF to finetune our model with both ITC(Contrastive Loss) and ITC(Sigmoid Loss). I also tried ITC(Contrastive Loss) only and it showed lower performance. (Note: ITC(Contrastive Loss) is quite sensitive to "batch_size".)
• You may need to add corresponding loss functions and related code to our huggingface implementation.

[aretius]

Got it thanks for the info. Sure will add loss function and related code to fine-tune on my data. I had two more follow up questions

• Since ITC really is sensitive to batch size, what about just training with ITM does that work well?
• Also in one of your sections of paper its mentioned that you finetuned for flickr30k etc on CE/BCE loss. So does that mean ITC using CE loss?

[LooperXX]

Hello, thanks for your questions.

• just training with ITM is a simple and cheap way for image-text retrieval task, but it will get a lower result than ITM+ITC (if you tune well, as we do in BridgeTower). If I remember correctly, ITC < ITM < ITM+ITC. (But I may not tune "ITC only" very well, XD)
• for all image-text retrieval tasks, our final result & checkpoints are fine-tuned with ITC(Contrastive Loss) and ITM(Sigmoid Loss).