Why choose to train and finetune LLM for detection task instead of Vit or other mid layers?

[zzhanghub]

We speculate that the generalization of ViT from CLIP could be strong; however, training on a small amount of data might lead to catastrophic forgetting. Both the intermediate FC layer and LLM are involved in the training process. Yet, the training strategy remains an open and unresolved issue, and we welcome further discussion on this matter.

[CYF2000127]

May I ask how to finetune ViT in the code? I set the ViT needs grad but still cant finetune I dont know why. Thank you very much.