search

j-min / VL-T5

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)
https://arxiv.org/abs/2102.02779
MIT License
357 stars 56 forks source link

Captioning off-the-shelf #5

Open shrutijpalaskar opened 2 years ago

shrutijpalaskar commented 2 years ago

Hello,

I am using pre-trained VL-T5 to generate captions for Flickr30K images off-the-shelf i.e. without any finetuning. I modified the captioning scripts to predict directly. I observe very short captions through, almost like noun phrases. I am including some examples below. I have played with the '--gen_max_length' and '--num_beams' parameters but I still get very short outputs. Do you have any ideas why this may be happening? Or any suggestions for how to potentially generate longer captions?

Thank you in advance! Shruti

purple shirt
cutting cake
smiling
large group of people
skier
j-min commented 2 years ago

It's probably because the pretraining objective for text generation (span prediction) always involves short target text. I guess zero-shot captioning might now work well. You would need to tune the parameters at least slightly, through few-shot or full fine-tuning.