search

LLaVA-VL / LLaVA-NeXT

Apache License 2.0
2.52k stars 186 forks source link

Demo deployment issues #85

Closed pedrocolon93 closed 3 months ago

pedrocolon93 commented 3 months ago

Hi there! I clone the tensors here: git clone https://huggingface.co/lmms-lab/llava-next-interleave-7b And I do the setup as is in the readme (which needs an upgrade for gradio (pip install --upgrade gradio) and needs numpy==1.23.0) and when i do inference in gradio (with the examples) I get garbage. Is there anything I am missing? image

pedrocolon93 commented 3 months ago

As a side note also needs to install flash attention: pip install flash-attn

pedrocolon93 commented 3 months ago

As a second side note same thing for the -dpo model.

pedrocolon93 commented 3 months ago

FIxed this by cloning the repo and adding -qwen- in the name of the repo... Otherwise it loads some other Llava architecture which does not work.

pedrocolon93 commented 3 months ago

If loading in 4 bit, the line in builder.py kwargs["load_in_4bit"] = True needs to be commented.

HaoZhang534 commented 3 months ago

You should change the model path from llava-next-interleave-7b to llava-next-interleave-qwen-7b and try again.

pedrocolon93 commented 3 months ago

Thanks, and double check the: If loading in 4 bit, the line in builder.py kwargs["load_in_4bit"] = True needs to be commented and adding in the flash-attn dependency