search

kdexd / virtex

[CVPR 2021] VirTex: Learning Visual Representations from Textual Annotations
http://kdexd.xyz/virtex
MIT License
556 stars 61 forks source link

Fine tuning Virtex for image captioning #19

Open freeIsa opened 3 years ago

freeIsa commented 3 years ago

Hi there, I am aware that Virtex used image captioning as a pretraining task and not as the "final goal", but I was wondering whether one could go on fine-tuning the pretrained model (e.g. bicaptioning_R_50_L1_H2048) with additional COCOcaptions-like data in order to get an improved captioning model. Has anyone tried that or does anyone have any suggestion how to do it? Can any of the scripts in the repository be used/adapted for fine-tuning existing models? Thanks a lot! :)

bermeitinger-b commented 3 years ago

You could use the pretrained ResNet-50 and then, similar to the pretraining layout, add the Transformers back in. But now, you freeze ResNet-50 and use more decoder layers instead of one.