search

Ucas-HaoranWei / Vary-toy

Official code implementation of Vary-toy (Small Language Model Meets with Reinforced Vision Vocabulary)
565 stars 41 forks source link

请问new vision vocabulary weights是否指的是sam部分的权重？ #22

Closed whalefa1I closed 4 months ago

whalefa1I commented 4 months ago

目前我训练了opt模型，请问是否在训练qwen的过程中将https://github.com/Ucas-HaoranWei/Vary-toy/blob/febea60eec6b96e88cdbff4dfe542b55751b19d9/Vary-master/vary/model/vary_toy_qwen1_8.py#L27 的权重替换成：

from vary.model.vary_opt import varyOPTForCausalLM
model_name = './checkpoints/qwen-vary-opt/'
model = varyOPTForCausalLM.from_pretrained(model_name)

这里的权重就并且freeze就可以extract new vision vocabulary weights？

Ucas-HaoranWei commented 4 months ago

是的，注意判断是否导入正确

whalefa1I commented 4 months ago

在https://github.com/Ucas-HaoranWei/Vary-toy/blob/febea60eec6b96e88cdbff4dfe542b55751b19d9/Vary-master/vary/train/train_qwen_vary.py#L38 这后面加是不是保险点，在里面加载一直报那些权重都是重新被初始化的

whalefa1I commented 4 months ago

训练qwen的时候数据要整理成qwen vl的格式吗？