Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
BLIP2 not support text input for Q-former. But InstuctBLIP support it. In this project, I use blip2 qformer not instructblip one. If you use instructblip q-former, you can change this code into the lavis one.
https://github.com/Coobiw/MiniGPT4Qwen/blob/d13f9657614a6be7553c850b7f95b4c31832eeef/lavis/models/minigpt4qwen_models/minigpt4qwen.py#L110C18-L110C18 predefine :qformer_text_input = True lavis repo code :self.Qformer.resize_token_embeddings(len(self.tokenizer))
your code:raise NotImplementedError
Why do you change it like this?