search

DAMO-NLP-SG / VideoLLaMA2

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
Apache License 2.0
850 stars 58 forks source link

Cannot reproduce results on vllava datasets #81

Open williamium3000 opened 2 months ago

williamium3000 commented 2 months ago

Dear authors of VideoLLaMA2, Thanks for the great work. We tried to reproduce your results on vllava datasets using the latest version of the code. However, we observe a large discrepancy in the three test datasets.

Model MVBench Egoschema ActivityNet Avg
reported 45.5 42.2 47.6 45.1
reproduced 44.475 38.5 43.55 42.175

We directly use your code, and follow your instructions to download the vllava datasets as well as three test sets, i.e. MVBench, Egoschema, and ActivityNet.

Can you hint at how you achieved the average 45.1 results?

Best Yijiang

clownrat6 commented 2 months ago

Please adopt the new scripts to train our videollama2 under videollava settings. Previous scripts adopt another projector, which is not consistent with the projector of this experiment.

williamium3000 commented 2 months ago

I am using the lastest. Unless you guys updated within two days. My results were obatined with the code pulled last weekend. I use the pretrain.sh and finetune.sh in scripts/vllava

lixin4ever commented 2 months ago

Please adopt the new scripts to train our videollama2 under videollava settings. Previous scripts adopt another projector, which is not consistent with the projector of this experiment.

Hi Yijiang, my colleague may not state this clearly :joy: We updated the fine-tuning script this afternoon, please check the latest commit and launch your training jobs (on video-llava dataset) with the new script.

williamium3000 commented 2 months ago

Oh thanks!!! Sorry for the misunderstanding. I will try tonight.

williamium3000 commented 2 months ago
Hi @lixin4ever and @clownrat6, We have switched to connector-v35 but still cannot reproduce. The results are even lower than the first version. Model MVBench Egoschema ActivityNet Avg
reported 45.5 42.2 47.6 45.1
reproduced 44.475 38.5 43.55 42.175
reproduced connector-v35 43.5 35.38 41.48 40.12
williamium3000 commented 2 months ago

We attached all the json files generated here: config.json generation_config.json model.safetensors.index.json special_tokens_map.json tokenizer.json tokenizer_config.json trainer_state.json

williamium3000 commented 2 months ago

env we use: name: videollama channels:

zhuqiangLu commented 1 month ago

Hi, I am going to reproduce this experiment. May I ask how many gpus did you use and how many days it took to run?

lixin4ever commented 1 month ago

Hi @lixin4ever and @clownrat6, We have switched to connector-v35 but still cannot reproduce. The results are even lower than the first version.

Model MVBench Egoschema ActivityNet Avg reported 45.5 42.2 47.6 45.1 reproduced 44.475 38.5 43.55 42.175 reproduced connector-v35 43.5 35.38 41.48 40.12

Hi Yijiang, We found that the latest codebase, migrated from the older one (I.e., V1.0) to be better compatible with Qwen2 (and other recent LLMs), indeed suffers from performance degradation when switching the language decoder to Mistral. However, due to the lack of resources, we temporarily have no GPUs to further verify if the code migration leads to this issue. We will continue the verification in early October, please stay tuned.

lixin4ever commented 1 month ago

Hi, I am going to reproduce this experiment. May I ask how many gpus did you use and how many days it took to run?

Two A100/A800 nodes (i.e., 16 GPUs) for < 20 hours (pretraining + fine-tuning)

zhuqiangLu commented 1 month ago

Hi, I am going to reproduce this experiment. May I ask how many gpus did you use and how many days it took to run?

Two A100/A800 nodes (i.e., 16 GPUs) for < 20 hours (pretraining + fine-tuning)

Oh, that is much faster than I thought, thank you. Are you training full model or using LoRA?

zhuqiangLu commented 1 month ago

Hi, I am going to reproduce this experiment. May I ask how many gpus did you use and how many days it took to run?

Two A100/A800 nodes (i.e., 16 GPUs) for < 20 hours (pretraining + fine-tuning)

Also, I just tried using 8xA100 for the pretraining stage, it estimates the pretraining stage will take 48 hours, could you please clarify that the pretraining stage should include both valley and llavaimage dataset?

lixin4ever commented 1 month ago

Yes, both Valley and LLaVA-Image should be included.

Regarding the time cost, I just checked the pretraining log of one run and it took around 8 hours on 2 A800 nodes (i.e., 16 80G-A800s).

zhuqiangLu commented 1 month ago

Yes, both Valley and LLaVA-Image should be included.

Regarding the time cost, I just checked the pretraining log of one run and it took around 8 hours on 2 A800 nodes (i.e., 16 80G-A800s).

Thank you for your response, may I ask for the checkpoint of the model trained on valley dataset? I am keen to see how it performs on my custom dataset.

williamium3000 commented 1 month ago

Hi, I am going to reproduce this experiment. May I ask how many gpus did you use and how many days it took to run?

I use 8 a800 80G GPUs. local and global batch size follows the scripts.

zhuqiangLu commented 1 month ago

Hi there, I have finished my experiment to reproduce the result with vllava. My result seems consistent with the reported result.

Model MVBENCH Egoschema ActivityNet
reported 45.5 42.2 47.6
reproduce 45.65 43.5 -

I will update the performance of ActivityNet later today, the dataset is still downloading.

I have made TWO modifications:

  1. Finetuning with LoRA, r = 128, alpha = 256
  2. Vllava dataset is downloaded from here

@williamium3000 I compared my trainer_state.json with yours, and noticed your grad_norms are much higher than mine. In addition, my loss values are slightly lower as well.

Here is my trainer_state.json

I ran my experiment on a 8 x L40s machine.

zhuqiangLu commented 1 month ago
Screenshot 2024-09-21 at 4 28 00 pm

Here is Egoschema evolution result from Kaggle.

williamium3000 commented 1 month ago

@zhuqiangLu How to train with lora? Do you mean that you first pretrain on the valley pretrain dataset and then use lora to fine-tune on sft dataset?

zhuqiangLu commented 1 month ago

@zhuqiangLu How to train with lora? Do you mean that you first pretrain on the valley pretrain dataset and then use lora to fine-tune on sft dataset?

That is right. The pretraining is done on valley dataset with the official script. To enable lora with SFT, simply add --enable_lora --lora_r 128 --lora_alpha 256 to the fine-tuning script.


torchrun 
--nnodes $WORLD_SIZE \
    --nproc_per_node $NPROC_PER_NODE \
    --master_addr=$MASTER_ADDR \
    --master_port=$MASTER_PORT \
    --node_rank $RANK \
    videollama2/train_flash_attn.py \
    --lora_enable True \
    --lora_r 128 \
    --lora_alpha 256 \
    --deepspeed scripts/zero3.json \
    --model_type videollama2 \
    --model_path mistralai/Mistral-7B-Instruct-v0.2 \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --mm_projector_type stc_connector_v35 \
    --pretrain_mm_mlp_adapter ${OUTP_DIR}/${WANDB_PROJECT}/pretrain_${RUN_NAME}/mm_projector.bin \
    --data_path   ${DATA_DIR}/videollava_sft/videochatgpt_llavaimage_tune.json \
    --data_folder ${DATA_DIR}/videollava_sft/ \
    --mm_vision_select_layer -2 \
    --image_aspect_ratio pad \
    --num_frames 8 \
    --bf16 True \
    --tf32 True \
    --fp16 False \
    --output_dir ${OUTP_DIR}/${WANDB_PROJECT}/finetune_${RUN_NAME} \
    --num_train_epochs 1 \
    --per_device_train_batch_size $LOCAL_BATCH_SIZE \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps $GRADIENT_ACCUMULATION_STEPS \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 500 \
    --save_total_limit 99 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --report_to tensorboard \
    --run_name $RUN_NAME \```
williamium3000 commented 1 month ago

I believe their original result is produced by full fine-tuning. I have not tried lora though.

clownrat6 commented 1 month ago

Using v1.0 tag version codebase, I retrain the VideoLLaMA with tcadapterv35 on videollava dataset. It seems that v1.0 version can reproduce the results in paper.

git checkout v1.0
bash scripts/vllava/stc/pretrain.sh
bash scripts/vllava/stc/finetune.sh
The results are: Model MVBench Egoschema
reported 45.5 42.2
reproduce (v1.0 tag) 44.1 41.6

The training loss curve is: image

The finetuning loss curve is: image

The evaluation results are: image image

williamium3000 commented 1 month ago

I follow @zhuqiangLu and try to reproduce his result with lora.

Model MVBENCH Egoschema ActivityNet
reported 45.5 42.2 47.6
reproduce by @zhuqiangLu 45.65 43.5 -
my lora 43.625 41.62 44.17

Despite it's higher than the full-finetuning, it's still behind the reported or the results by @zhuqiangLu

zhuqiangLu commented 4 weeks ago

I was thinking could it be the dataset? My vllava dataset was downloaded from huggingface, then I simply use the jsons provided by videollama2.

clownrat6 commented 4 weeks ago

I follow @zhuqiangLu and try to reproduce his result with lora.

Model MVBENCH Egoschema ActivityNet reported 45.5 42.2 47.6 reproduce by @zhuqiangLu 45.65 43.5 - my lora 43.625 41.62 44.17 Despite it's higher than the full-finetuning, it's still behind the reported or the results by @zhuqiangLu

Could you please attempt v1.0 tag version code? You can checkout this version by git checkout v1.0, which will achieve higher performance than main branch.

williamium3000 commented 3 weeks ago

Yeah, we have observed a similar trend. v1.0 does seem better. However, I wondering how @zhuqiangLu achieved such high result. @zhuqiangLu Did you use v1.0 as well?

zhuqiangLu commented 3 weeks ago

It was default branch, but now I have no available GPUs for this training, I will update when I finish training with the v1.0 branch.