search

sshh12 / multi_token

Embed arbitrary modalities (images, audio, documents, etc) into large language models.
Apache License 2.0
158 stars 8 forks source link

Finetuning already trained model #5

Closed Aniketto16 closed 6 months ago

Aniketto16 commented 6 months ago

can I finetune this model ? https://huggingface.co/sshh12/Mistral-7B-LoRA-Multi-VisionCLIPPool-LLAVA

If so how to do it, I am in a hurry so please reply as soon as possible! Thank you!

sshh12 commented 6 months ago

Sure, you can just follow the instructions for https://github.com/sshh12/multi_token?tab=readme-ov-file#finetuning but use the the pre-existing projector from https://huggingface.co/sshh12/Mistral-7B-LoRA-Multi-VisionCLIPPool-LLAVA/blob/main/non_lora_trainables.bin

Aniketto16 commented 6 months ago

Thank you so much, I understand!