search

LLaVA-VL / LLaVA-NeXT

Apache License 2.0
2.77k stars 223 forks source link

Inferencing Finetuned model #155

Open rfoxes opened 2 months ago

rfoxes commented 2 months ago

Hello, I finetuned LLaVA onevision with Qwen2-7B. In the finetuning script, I set it to finetune just the adapter. When I am trying to inference my model, I am using the example code given in the repo, and I pass in the pretrained model with no model_base, and the name as 'llava_qwen' to the load_pretrained_model function. However, I keep getting this error:

OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory ../checkpoints/llavanext-Qwen-7B-siglip-so400m-patch14-384-onevision-20K.

I am not sure how to fix this. Can anyone help with this error?

rfoxes commented 2 months ago

Hello, I was able to fix the error by downloading the safe tensors from the huggingface repo, but now the output of the model for an image is ['!!!!!!!!!!!']. Is this an error that occurred when fine-tuning the model, or something wrong with inferencing it?

sc268 commented 2 months ago

got the same output ['!!!!'] @rfoxes any suggestions? Thanks!

rfoxes commented 1 month ago

I downloaded the projector files, and it changed the output, but it is still unreadable.

Here is an example ["ADM<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<oppable<\u6cbb\u5b890\u5b9a\u5f8b (licted \u59f9 - xhttp<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u5c42\u51fa<\u6cbb\u5b890\u5b9a\u5f8b<\",-<\",-<\"class<\",-<)){\n\n< multer<\",-<\",-<.Physics<.Physics<\",-<\",-<\",-<\u9648\u5217<\u9648\u5217<\",-<\",-<chnitt\u4ea6 ++;\r\n\r\n (\u781c<..."]