search

Coobiw / MPP-LLaVA

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
349 stars 19 forks source link

Code confusion #2

Closed abbhay closed 4 months ago

abbhay commented 10 months ago

https://github.com/Coobiw/MiniGPT4Qwen/blob/d13f9657614a6be7553c850b7f95b4c31832eeef/lavis/models/minigpt4qwen_models/minigpt4qwen.py#L110C18-L110C18 predefine ：qformer_text_input = True lavis repo code ：self.Qformer.resize_token_embeddings(len(self.tokenizer))

your code：raise NotImplementedError

Why do you change it like this?

Coobiw commented 10 months ago

BLIP2 not support text input for Q-former. But InstuctBLIP support it. In this project, I use blip2 qformer not instructblip one. If you use instructblip q-former, you can change this code into the lavis one.