[Question] int-8 bits for instruct tuning

[breezedeus]

Question

Many thanks to the authors for this very good work.

I'm trying to run the instruct tuning part on one 3090 GPU, using the int-8 mode:

    python llava/train/train.py \
    --model_name_or_path lmsys/vicuna-7b-v1.3 \
    --version v1 \
    --data_path ./LLaVA-Instruct-150K/llava_instruct_80k.json \
    --image_folder ./coco/train2017 \
    --vision_tower openai/clip-vit-large-patch14 \
    --pretrain_mm_mlp_adapter checkpoints/llava-pretrain-vicuna-7b-v1.3/mm_projector.bin \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --bf16 True \
    --bits 8 \
    --output_dir ./checkpoints/llava-vicuna-v1-3-7b-finetune-tmp \
    --num_train_epochs 1 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1000 \
    --max_steps -1 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to wandb

The above command can finish the training and save the model successfully. The following is the loss curve:

But when I use the following command to make predictions using the trained model, the result is very bad:

    python llava/eval/run_llava.py \
    --model-path ./checkpoints/llava-vicuna-v1-3-7b-finetune-tmp \
    --image-file images/llava_logo.png \
    --query "describe the image"

It's the result:

chevlocchevchevchevlocchevlocchevchevlocchevchevlocchevlocchevlocchevloclocchevlocchevchevchevlocchevloclocchevlocchevchevchevlocchevlocchevloclocchevlocchevchevloclocchevlocch
evlocchevlocchevchevlocchevloclocloclocchevloclocchevloclocchevchevchevlocchevlocchevchevchevlocchevchevlocchevlocchevlocchevlocchevchevloclocchevchevchevchevchevlocchevloclocc
hevlocchevlocchevchevloclocchevchevchevchevlocloclocloclocchev Хронологијасняchevlocchevchevchevlocchevchevlocchevlocchevchevchevlocchevchevchevlocchevlocchevchevchevlocchevmer
kсняchevchevlocchev Хронологијаchevсняchevmerkchevchevchevchevchev Хронологијаchevmerkchevchevсняchevchevchevсняmerkmerklocchevchevchevchevchevchevchevmerklocmerkchev Хронологи
 ...

Does anyone know why? Is it because I'm using the model incorrectly? Or is this model less effective in int-8 mode?

Thanks much.

[haotian-liu]

Hi, I have not tried or verified the full model finetuning using int8 training. Int8/Int4 are mainly designed for quantized LoRA training. Please check out the instructions here.

[breezedeus]

Hi, I have not tried or verified the full model finetuning using int8 training. Int8/Int4 are mainly designed for quantized LoRA training. Please check out the instructions here.

@haotian-liu Thanks for your suggestion. I know there were problems with the previous training, so I changed to lora+8-bits training as you suggested. Below is my instruct tuning loss curve:

The results generated from the model are not very satisfactory. For instance, I want the model to describe the llava logo image:

python llava/eval/run_llava.py \
--model-path ./checkpoints/llava-vicuna-v1-3-7b-finetune-lora \
--model-base lmsys/vicuna-7b-v1.3 \
    --load-8bit \
    --image-file images/llava_logo.png \
    --query "describe the image"

The following are generated responses：

The image is a close-up of a person's face, with a blurred background. The person's eyes are open, and their facial expression appears to be one of concentration or focus. The
blurred background adds a sense of depth and focus to the image, making the person's face the primary point of interest. The overall effect is a visually striking and engaging
portrait that captures the viewer's attention.

Would you mind providing your Lora instruct tuning logs? Thanks much.