Isaachhh
commented
1 week ago For another vision tower or projector, you can import what you like. Pay attention to multimodal_encoder and multimodal_projector. You need to add the code of class and modify the build function.
For combining multiple vision features, you also need to modify the architecture of Bunny (something like vision_tower_list) and encode_image function and etc.
Generally, you need to pre-train and fine-tune by yourself. Under some circumstances, you may start from our released weights.
Why0912
请问是否支持修改视觉模块或融合多个主干的视觉表征？ 如果进行修改或融合，是否需要重新进行pre_train来获得相应的projector权重？ 或是如何对projector进行修改？