search

PKU-YuanGroup / MoE-LLaVA

Mixture-of-Experts for Large Vision-Language Models
https://arxiv.org/abs/2401.15947
Apache License 2.0
1.9k stars 121 forks source link

RuntimeError: mat1 and mat2 must have the same dtype #63

Open Crystalxd opened 5 months ago

Crystalxd commented 5 months ago

logs that inferred custom trained model (LanguageBind + Qwen14B LLM).

(moellava) root@ps:/code/MoE-LLaVA# CUDA_VISIBLE_DEVICES=0 python predict.py 
[2024-03-18 02:02:14,276] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/opt/conda/envs/moellava/lib/python3.10/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
  warnings.warn(
/opt/conda/envs/moellava/lib/python3.10/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
  warnings.warn(
/opt/conda/envs/moellava/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
The model is automatically converting to bf16 for faster inference. If you want to disable the automatic precision, please manually add bf16/fp16/fp32=True to "AutoModelForCausalLM.from_pretrained".
projector_type: mlp2x_gelu
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:24<00:00,  4.07s/it]
Some weights of the model checkpoint at /output/llava-qwen14/checkpoint-200/ were not used when initializing LlavaQWenForCausalLM: ['transformer.image_tower.image_tower.embeddings.class_embedding', 'transformer.image_tower.image_tower.embeddings.patch_embedding.weight', 'transformer.image_tower.image_tower.embeddings.position_embedding.weight', 'transformer.image_tower.image_tower.encoder.layers.0.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.0.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.0.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.0.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.0.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.0.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.0.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.0.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.0.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.1.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.1.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.1.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.1.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.1.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.1.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.1.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.1.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.1.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.10.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.10.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.10.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.10.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.10.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.10.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.10.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.10.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.10.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.11.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.11.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.11.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.11.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.11.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.11.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.11.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.11.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.11.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.12.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.12.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.12.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.12.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.12.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.12.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.12.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.12.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.12.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.13.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.13.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.13.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.13.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.13.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.13.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.13.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.13.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.13.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.14.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.14.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.14.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.14.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.14.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.14.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.14.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.14.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.14.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.15.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.15.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.15.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.15.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.15.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.15.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.15.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.15.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.15.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.16.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.16.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.16.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.16.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.16.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.16.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.16.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.16.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.16.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.17.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.17.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.17.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.17.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.17.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.17.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.17.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.17.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.17.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.18.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.18.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.18.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.18.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.18.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.18.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.18.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.18.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.18.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.19.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.19.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.19.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.19.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.19.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.19.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.19.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.19.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.19.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.2.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.2.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.2.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.2.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.2.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.2.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.2.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.2.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.2.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.20.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.20.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.20.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.20.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.20.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.20.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.20.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.20.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.20.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.21.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.21.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.21.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.21.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.21.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.21.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.21.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.21.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.21.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.22.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.22.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.22.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.22.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.22.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.22.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.22.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.22.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.22.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.23.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.23.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.23.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.23.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.23.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.23.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.23.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.23.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.23.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.3.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.3.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.3.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.3.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.3.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.3.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.3.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.3.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.3.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.4.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.4.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.4.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.4.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.4.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.4.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.4.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.4.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.4.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.5.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.5.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.5.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.5.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.5.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.5.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.5.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.5.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.5.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.6.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.6.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.6.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.6.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.6.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.6.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.6.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.6.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.6.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.7.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.7.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.7.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.7.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.7.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.7.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.7.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.7.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.7.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.8.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.8.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.8.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.8.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.8.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.8.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.8.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.8.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.8.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.9.layer_norm1.bias', 'transformer.image_tower.image_tower.encoder.layers.9.layer_norm1.weight', 'transformer.image_tower.image_tower.encoder.layers.9.layer_norm2.bias', 'transformer.image_tower.image_tower.encoder.layers.9.layer_norm2.weight', 'transformer.image_tower.image_tower.encoder.layers.9.mlp.fc1.bias', 'transformer.image_tower.image_tower.encoder.layers.9.mlp.fc1.weight', 'transformer.image_tower.image_tower.encoder.layers.9.mlp.fc2.bias', 'transformer.image_tower.image_tower.encoder.layers.9.mlp.fc2.weight', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.k_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.k_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.out_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.out_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.q_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.q_proj.weight', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.v_proj.bias', 'transformer.image_tower.image_tower.encoder.layers.9.self_attn.v_proj.weight', 'transformer.image_tower.image_tower.post_layernorm.bias', 'transformer.image_tower.image_tower.post_layernorm.weight', 'transformer.image_tower.image_tower.pre_layrnorm.bias', 'transformer.image_tower.image_tower.pre_layrnorm.weight']
- This IS expected if you are initializing LlavaQWenForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LlavaQWenForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
LlavaQWenForCausalLM(
  (transformer): LlavaQWenModel(
    (wte): Embedding(152064, 5120)
    (drop): Dropout(p=0.0, inplace=False)
    (rotary_emb): RotaryEmbedding()
    (h): ModuleList(
      (0-39): 40 x QWenBlock(
        (ln_1): RMSNorm()
        (attn): QWenAttention(
          (c_attn): Linear(in_features=5120, out_features=15360, bias=True)
          (c_proj): Linear(in_features=5120, out_features=5120, bias=False)
          (attn_dropout): Dropout(p=0.0, inplace=False)
        )
        (ln_2): RMSNorm()
        (mlp): QWenMLP(
          (w1): Linear(in_features=5120, out_features=13696, bias=False)
          (w2): Linear(in_features=5120, out_features=13696, bias=False)
          (c_proj): Linear(in_features=13696, out_features=5120, bias=False)
        )
      )
    )
    (ln_f): RMSNorm()
    (image_tower): LanguageBindImageTower()
    (mm_projector): build_projector(
      (image_spatial_proj): Sequential(
        (0): Linear(in_features=1024, out_features=5120, bias=True)
        (1): GELU(approximate='none')
        (2): Linear(in_features=5120, out_features=5120, bias=True)
      )
      (video_patch_proj): Identity()
      (video_spatial_proj): Identity()
      (video_temproal_proj): Identity()
      (video_global_proj): Identity()
    )
  )
  (lm_head): Linear(in_features=5120, out_features=152064, bias=False)
)
/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
/opt/conda/envs/moellava/lib/python3.10/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
  warnings.warn(
ASSISTANT: What is the man in the picture doing?
2024-03-18 02:02:45.831 | WARNING  | __main__:main:38 - ==================
tensor([[    32,   6236,   1948,    264,  22208,   1196,    323,    458,  20443,
          11229,  17847,     13,    576,  17847,   6696,  10950,     11,  11682,
             11,    323,  47787,  11253,    311,    279,   1196,    594,   4755,
             13,  13872,     25,    220,   -200,    198,  45930, 101047, 102015,
          18493, 106428,     30,  35560,   3846,   2821,     25]],
       device='cuda:0')
tensor([[[[-1.7891, -1.7891, -1.7891,  ..., -1.7930, -1.7930, -1.7891],
          [-1.7627, -1.7676, -1.7627,  ..., -1.7686, -1.7559, -1.7637],
          [-1.7461, -1.7480, -1.7471,  ..., -1.7490, -1.7363, -1.7520],
          ...,
          [-1.7285, -1.7344, -1.6748,  ..., -1.7461, -1.7266, -1.7402],
          [-1.7686, -1.7510, -1.7715,  ..., -1.7949, -1.7402, -1.7734],
          [-1.7832, -1.7910, -1.7852,  ..., -1.7891, -1.7900, -1.7930]],

         [[-1.7539, -1.7500, -1.7422,  ..., -1.7412, -1.7432, -1.7539],
          [-1.7002, -1.7051, -1.7021,  ..., -1.7422, -1.7246, -1.7119],
          [-1.6758, -1.6797, -1.6826,  ..., -1.6777, -1.6650, -1.6807],
          ...,
          [-1.6445, -1.6914, -1.6289,  ..., -1.7041, -1.6758, -1.6982],
          [-1.7168, -1.7119, -1.7451,  ..., -1.7383, -1.7158, -1.7324],
          [-1.7432, -1.7471, -1.7451,  ..., -1.7490, -1.7529, -1.7510]],

         [[-1.4814, -1.4814, -1.4814,  ..., -1.4834, -1.4834, -1.4834],
          [-1.4434, -1.4473, -1.4463,  ..., -1.4082, -1.3926, -1.3936],
          [-1.3838, -1.3867, -1.3867,  ..., -1.3975, -1.3994, -1.3994],
          ...,
          [-1.4268, -1.4434, -1.3691,  ..., -1.4326, -1.4082, -1.4443],
          [-1.4775, -1.4551, -1.4697,  ..., -1.4756, -1.4756, -1.4590],
          [-1.4629, -1.4678, -1.4668,  ..., -1.4697, -1.4736, -1.4805]]]],
       device='cuda:0', dtype=torch.float16)
2024-03-18 02:02:45.843 | WARNING  | __main__:main:41 - ==================
++++++++++++++++
tensor([[[-0.9653,  0.5757, -1.1807,  ...,  1.2803, -0.7188, -0.8818],
         [ 0.4961,  2.7051,  0.0115,  ...,  0.6382, -0.6060, -0.5703],
         [ 0.1807,  0.8447,  0.4824,  ...,  1.0771, -0.0136, -1.2354],
         ...,
         [ 0.6416,  1.0879, -0.5303,  ...,  1.1309, -0.9102, -0.0253],
         [ 0.3801,  3.1152, -0.9663,  ..., -0.0643, -0.4917,  1.3672],
         [-0.8354,  0.7363, -1.6709,  ...,  1.4736, -0.3210, -0.8779]]],
       device='cuda:0', dtype=torch.float16)
image_feature_shape: torch.Size([1, 256, 1024])
Traceback (most recent call last):
  File "/code/MoE-LLaVA/predict.py", line 57, in <module>
    main()
  File "/code/MoE-LLaVA/predict.py", line 44, in main
    output_ids = model.generate(
  File "/code/MoE-LLaVA/moellava/model/language_model/qwen/modeling_qwen.py", line 1260, in generate
    return super().generate(
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/transformers/generation/utils.py", line 1520, in generate
    return self.sample(
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/transformers/generation/utils.py", line 2617, in sample
    outputs = self(
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/code/MoE-LLaVA/moellava/model/language_model/llava_qwen.py", line 147, in forward
    ) = self.prepare_inputs_labels_for_multimodal(
  File "/code/MoE-LLaVA/moellava/model/llava_arch.py", line 458, in prepare_inputs_labels_for_multimodal
    image_features_minibatch = self.encode_images(images_minibatch)  # [mini_b, l, c]
  File "/code/MoE-LLaVA/moellava/model/llava_arch.py", line 155, in encode_images
    image_features = self.get_model().mm_projector.forward_image(image_features)
  File "/code/MoE-LLaVA/moellava/model/multimodal_projector/builder.py", line 140, in forward_image
    return self.image_spatial_proj(image_feature)
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/moellava/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype

pretrain.sh

JSON_FOLDER="/data/llava_pt/json"
IMAGE_FOLDER="/data"
# cd ~/MoE-LLaVA
# HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1
CUDA_VISIBLE_DEVICES=0,1,2,3 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 deepspeed moellava/train/train_mem.py \
    --deepspeed ./scripts/zero2.json \
    --model_name_or_path /model/Qwen-14B \
    --version plain \
    --data_path ${JSON_FOLDER}/llava_image_.json \
    --image_folder ${IMAGE_FOLDER} \
    --image_tower /model/LanguageBind/LanguageBind_Image \
    --image_projector_type mlp2x_gelu \
    --tune_mm_mlp_adapter True \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --bf16 True \
    --output_dir /output/llavaqwen-14b-pretrain \
    --num_train_epochs 1.5 \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 2 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 500 \
    --save_total_limit 1 \
    --learning_rate 1e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 4096 \
    --gradient_checkpointing True \
    --dataloader_num_workers 8 \
    --lazy_preprocess True \
    --report_to tensorboard \
    --cache_dir "./cache_dir"

predict.py

import torch
from PIL import Image
from moellava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from moellava.conversation import conv_templates, SeparatorStyle
from moellava.model.builder import load_pretrained_model
from moellava.utils import disable_torch_init
from moellava.mm_utils import tokenizer_image_token, get_model_name_from_path, KeywordsStoppingCriteria
from loguru import logger as log

def main():
    disable_torch_init()
    # image = 'moellava/serve/examples/extreme_ironing.jpg'
    # inp = 'What is unusual about this image?'
    image = '/data/lrv_tune/images/2371990.jpg'
    inp = 'What is the man in the picture doing?'
    model_path = '/output/llava-qwen14/checkpoint-200/'  # choose a model
    device = 'cuda'
    load_4bit, load_8bit = False, False
    model_name = get_model_name_from_path(model_path)
    tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device)
    image_processor = processor['image']
    conv_mode = "qwen"  # phi or qwen or stablelm
    conv = conv_templates[conv_mode].copy()
    roles = conv.roles

    image_tensor = image_processor.preprocess(Image.open(image).convert('RGB'), return_tensors='pt')['pixel_values'].to(model.device, dtype=torch.float16)

    print(f"{roles[1]}: {inp}")
    inp = DEFAULT_IMAGE_TOKEN + '\n' + inp
    conv.append_message(conv.roles[0], inp)
    conv.append_message(conv.roles[1], None)
    prompt = conv.get_prompt()
    input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()
    stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
    keywords = [stop_str]
    stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
    log.warning("==================")
    print(input_ids)
    print(image_tensor)
    log.warning("==================")

    with torch.inference_mode():
        output_ids = model.generate(
            input_ids,
            images=image_tensor,
            do_sample=True,
            temperature=0.2,
            max_new_tokens=1024,
            use_cache=True,
            stopping_criteria=[stopping_criteria])

    outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:], skip_special_tokens=True).strip()
    print(outputs)

if __name__ == '__main__':
    main()
    '''
    deepspeed predict.py
    '''

finetune.sh

#!/bin/bash

JSON_FOLDER="/data/lrv_tune/json"
IMAGE_FOLDER="/data"
cd /code/MoE-LLaVA
CUDA_VISIBLE_DEVICES=0,1,2,3 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 deepspeed moellava/train/train_mem.py \
    --deepspeed ./scripts/zero2_offload.json \
    --model_name_or_path /model/Qwen-14B \
    --version qwen \
    --data_path ${JSON_FOLDER}/chinese_lrv_tune_50k.json \
    --image_folder ${IMAGE_FOLDER} \
    --image_tower /model/LanguageBind/LanguageBind_Image \
    --image_projector_type mlp2x_gelu \
    --pretrain_mm_mlp_adapter /output/llavaqwen-14b-pretrain/mm_projector.bin \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --group_by_modality_length True \
    --bf16 True \
    --output_dir /output/llava-qwen14 \
    --num_train_epochs 2.3 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 2 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 200 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 4096 \
    --gradient_checkpointing True \
    --dataloader_num_workers 16 \
    --lazy_preprocess True \
    --report_to tensorboard \
    --cache_dir "./cache_dir"