search

BAAI-DCAI / Bunny

A family of lightweight multimodal models.
Apache License 2.0
934 stars 69 forks source link

About continuous_training #137

Open Wintoplay opened 6 days ago

Wintoplay commented 6 days ago

With continuous_training on bunny VLM, do we still need to specify vision_tower path?

If we do point to siglip path, will it use that untrained weight instead of vision_tower that comes with bunny vlm I downloaded from huggingface?

What should I specify?

Isaachhh commented 6 days ago

Yes, it's still needed to specify --vision_tower to the path to huggingface/siglip-so400m-patch14-384.

But the vision tower of Bunny models would be used because "continuous_training": true in /path/to/merged_model/config.json.

https://github.com/BAAI-DCAI/Bunny?tab=readme-ov-file#continuous--fine-tuning

Wintoplay commented 6 days ago

do it work fine with this model too? https://huggingface.co/scb10x/llama-3-typhoon-v1.5-8b-vision-preview since it is trained by Bunny method

Isaachhh commented 6 days ago

I think so.

But also check https://github.com/BAAI-DCAI/Bunny/issues/130.