Closed rringham closed 11 months ago
Please add a "lora" in the model path so that it knows that it is a lora trained weights. We're optimizing the parsing strategy, but currently this should solve your problem. And please do not rename the file.
--output_dir ./checkpoints/llava-v1.5-7b-mytask-lora \
--model-path ./checkpoints/llava-v1.5-7b-mytask-lora \
Thank you, that worked for me. Much appreciated!
Question
Hi - I'm trying to wrap my head around how I'd fine-tune LLaVA for a specific use case. As an experiment, I have 28k images that I've generated a
dataset.json
for (am just programmatically generating text descriptions based on already known classes).My somewhat vague understanding is that I should be able to fine-tune llava-v1.5-7b by doing something like the following (and nothing more):
However, when I go to use (see following command) it will complain that
mm_projection.bin
is missing.I can get the above command to work if I manually rename
./checkpoints/llava-v1.5-7b/non_lora_trainables.bin
to be./checkpoints/llava-v1.5-7b/mm_projection.bin
, but the responses I see written toanswers.json
don't really appear as though they have been fine-tuned on my dataset - they look like the kinds of responses I get from the stockliuhaotian/llava-v1.5-7b
model.It leaves doubt in my mind that I'm actually correctly fine-tuning. Am I missing any key steps?