DINOv2 和 SigLIP 融合

BAAI-DCAI / Bunny

A family of lightweight multimodal models.

Apache License 2.0

799 stars 61 forks source link

DINOv2 和 SigLIP 融合 #77

Closed CanvaChen closed 2 months ago

CanvaChen commented 2 months ago

有模型采用 DINOv2 和 SigLIP 融合的视觉表征，通过将两个视觉编码器的输出拼接在一起送入投影器，模型能够更好的捕捉到 SigLIP 带来的高层次的语义特征和 DINOv2 提取的低层次的细粒度图像特征。有试过这样是否能使Bunny效果更好吗？

Isaachhh commented 2 months ago

Thank you for your interest, and my apologies for not exploring this strategy. Should you wish to proceed, feel free to try to combine multiple vision towers.