Finetune on other works/datasets

[mrluin]

Hello,

Thanks for your code and novel solution to visual prompt tuning!

If i want to finetune the pretrained model you provided on other works (assumpt that i have standard training pairs), what i understand the manucrafted input-label pairs should be:

1. stitch training pairs into 2x2 grids, as label.
2. leave the bottom-right image zero, as input.
3. Finetune!

Is that right? Could you please tell me how can i do it?

[amirbar]

You can leave zeros, or even keep the bottom right part masked.