LukeForeverYoung
commented
1 year ago Sorry, visual abstractor should not be trained during the instruction finetuning stage. The code provided in this repository has been updated, and we will correct the description in the paper as soon as possible.
It was an early experimental attempt to train visual abstractor during instruction finetuning, but later we found that setting requires_grad to False can achieve similar performance.
文中 Fig2 里 stage2 画的是Abstractor冻住了，但是论文描述和代码里似乎都是训练的？