Closed Ace-blue closed 1 month ago
Hello,
Thanks for your comment.
According to your info, the clip image encoder used was 336 size, but the token length of image was set to 256 (for vit-l-14-224), Please modify the value of the key "image_token_len" in the config to 576 (for vit-l-14-336).
Or, we will update the code that supports LLaVA1.5 (vicuna1.1) in several weeks.
for example
model:
arch: 'ullava_core'
llm_path: '/home/zhangzy/data3/VLM/zoo/vicuna-7b-v1.1'
vision_encoder: '/home/zhangzy/data3/VLM/zoo/clip-vit-large-patch14-336'
vision_hidden_layer: -2
projector_from_scratch: true
conv_type: 'conv_simple'
task:
type: image_text_pretrain
collator_type: 'image_collator'
processor:
clip_image:
path: '/home/zhangzy/data3/VLM/zoo/clip-vit-large-patch14-336'
image_size: 336
dataset:
llava_cc3m:
data_type: 'image'
image_token_len: 576
build_info:
anno_dir: '/home/zhangzy/data3/VLM/u-LLaVA/dataset_zoo/LLaVA-CC3M-Pretrain-595K/chat.json'
image_dir: '/home/zhangzy/data3/VLM/zoo/ullava_image'
portion: 1.0
vis_processor: 'clip_image'
Thanks for your timely response.
I've tried setting the key "image_token_len" to 576 (for vit-l-14-336) or replacing vit-l-14-336 with vit-l-14. But the bug segmentation fault (core dumped) still occurs.
Could you provide more information about the environment or configuration?
(老哥, 这个bug 卡了我几天了, 换机器/重启的招数都试了, 就很奇怪, 能不能救救弟弟😭😭
Here is my shell output :
python train_ullava_core.py --cfg_path ./configs/train/ullava_core_stage1.yaml 19:42:20 ====== Model Attributes ====== { "arch": "ullava_core", "conv_type": "conv_simple", "llm_path": "model_zoo/vicuna_7b_v1.1", "projector_from_scratch": true, "vision_encoder": "model_zoo/clip-vit-large-patch14", "vision_hidden_layer": -2 } ====== Task Attributes ====== { "collator_type": "image_video_collator", "type": "image_text_pretrain" } ====== Processor Attributes ====== { "clip_image": { "image_size": 224, "path": "model_zoo/clip-vit-large-patch14" } } ====== Train Dataset Attributes ====== ======== llava_cc3m ======= { "build_info": { "anno_dir": "./dataset_zoo/LLaVA-CC3M-Pretrain-595K/chat.json", "image_dir": "./dataset_zoo/LLaVA-CC3M-Pretrain-595K/ullava_image", "portion": 1.0 }, "data_type": "image", "image_token_len": 256, "vis_processor": "clip_image" } ====== Evaluate Dataset Attributes ====== [2024-07-24 19:42:35,532] Loading Tokenizer [2024-07-24 19:42:35,563] Initializing uLLaVA Core Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:58<00:00, 59.05s/it] Some weights of UllavaCoreForCausalLM were not initialized from the model checkpoint at model_zoo/vicuna_7b_v1.1 and are newly initialized: ['vision_encoder.vision_model.encoder.layers.15.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.14.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.1.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.19.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.4.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.12.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.16.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.21.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.23.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.11.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.23.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.2.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.15.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.14.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.6.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.17.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.20.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.8.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.0.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.14.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.7.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.4.self_attn.k_proj.weight', 'vision_encoder.vision_model.embeddings.position_ids', 'vision_encoder.vision_model.encoder.layers.17.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.2.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.6.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.6.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.10.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.23.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.3.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.22.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.8.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.9.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.19.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.13.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.22.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.22.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.7.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.20.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.1.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.12.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.23.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.21.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.2.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.1.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.17.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.0.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.9.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.9.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.16.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.6.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.17.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.12.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.21.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.23.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.18.self_attn.q_proj.bias', 'vision_encoder.vision_model.embeddings.position_embedding.weight', 'vision_encoder.vision_model.encoder.layers.21.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.22.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.16.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.8.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.19.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.19.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.17.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.17.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.18.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.6.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.12.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.14.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.19.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.13.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.8.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.21.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.16.mlp.fc2.weight', 'vision_encoder.vision_model.pre_layrnorm.bias', 'vision_encoder.vision_model.encoder.layers.16.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.10.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.9.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.0.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.16.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.3.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.11.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.16.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.13.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.11.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.1.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.0.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.2.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.19.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.8.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.5.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.14.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.1.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.9.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.8.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.8.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.15.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.7.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.11.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.10.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.5.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.10.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.17.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.18.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.15.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.9.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.11.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.14.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.18.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.23.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.18.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.19.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.21.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.13.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.1.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.7.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.8.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.11.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.11.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.4.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.17.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.11.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.13.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.23.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.20.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.10.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.5.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.18.self_attn.v_proj.weight', 'vision_projector.weight', 'vision_encoder.vision_model.encoder.layers.17.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.6.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.16.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.5.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.22.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.2.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.3.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.9.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.21.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.7.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.15.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.9.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.19.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.18.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.4.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.20.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.12.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.8.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.17.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.6.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.17.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.8.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.14.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.12.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.2.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.11.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.21.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.11.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.6.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.19.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.22.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.2.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.5.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.13.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.3.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.22.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.14.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.4.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.7.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.11.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.23.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.4.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.7.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.22.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.5.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.20.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.9.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.10.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.23.mlp.fc1.bias', 'vision_encoder.vision_model.embeddings.patch_embedding.weight', 'vision_encoder.vision_model.encoder.layers.21.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.3.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.20.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.7.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.0.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.22.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.19.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.18.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.18.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.5.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.18.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.7.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.22.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.7.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.0.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.21.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.6.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.14.self_attn.q_proj.weight', 'vision_encoder.vision_model.pre_layrnorm.weight', 'vision_encoder.vision_model.encoder.layers.10.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.14.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.23.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.23.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.22.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.6.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.3.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.8.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.18.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.1.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.18.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.6.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.18.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.22.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.12.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.16.layer_norm1.weight', 'vision_encoder.vision_model.post_layernorm.weight', 'vision_encoder.vision_model.encoder.layers.11.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.21.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.8.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.17.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.19.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.19.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.9.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.18.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.9.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.17.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.20.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.8.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.5.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.7.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.20.self_attn.out_proj.weight', 'vision_projector.bias', 'vision_encoder.vision_model.encoder.layers.11.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.17.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.6.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.23.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.20.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.18.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.11.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.8.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.3.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.4.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.22.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.6.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.1.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.17.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.12.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.23.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.0.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.0.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.2.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.13.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.10.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.22.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.22.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.23.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.20.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.21.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.7.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.12.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.21.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.12.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.6.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.21.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.5.layer_norm1.bias', 'vision_encoder.vision_model.embeddings.class_embedding', 'vision_encoder.vision_model.encoder.layers.0.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.22.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.8.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.16.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.14.mlp.fc2.weight', 'vision_encoder.vision_model.post_layernorm.bias', 'vision_encoder.vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.21.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.19.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.16.self_attn.q_proj.weight', 'vision_encoder.vision_model.encoder.layers.4.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.9.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.4.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.10.layer_norm2.weight', 'vision_encoder.vision_model.encoder.layers.19.mlp.fc2.weight', 'vision_encoder.vision_model.encoder.layers.13.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.13.self_attn.out_proj.bias', 'vision_encoder.vision_model.encoder.layers.1.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.2.mlp.fc2.bias', 'vision_encoder.vision_model.encoder.layers.21.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.15.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.4.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.7.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.5.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.14.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.1.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.6.self_attn.out_proj.weight', 'vision_encoder.vision_model.encoder.layers.13.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.0.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.6.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.20.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.9.self_attn.k_proj.bias', 'vision_encoder.vision_model.encoder.layers.12.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.14.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.3.self_attn.q_proj.bias', 'vision_encoder.vision_model.encoder.layers.9.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.13.layer_norm1.bias', 'vision_encoder.vision_model.encoder.layers.23.layer_norm1.weight', 'vision_encoder.vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_encoder.vision_model.encoder.layers.19.self_attn.v_proj.weight', 'vision_encoder.vision_model.encoder.layers.23.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.19.self_attn.v_proj.bias', 'vision_encoder.vision_model.encoder.layers.14.self_attn.k_proj.weight', 'vision_encoder.vision_model.encoder.layers.11.layer_norm2.bias', 'vision_encoder.vision_model.encoder.layers.7.mlp.fc1.bias', 'vision_encoder.vision_model.encoder.layers.2.self_attn.q_proj.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Using pad_token, but it is not set yet. [2024-07-24 19:45:48,941] LLaMA model, Loading CLIP Vision Encoder Some weights of the model checkpoint at model_zoo/clip-vit-large-patch14 were not used when initializing CLIPVisionModel: ['text_model.encoder.layers.4.self_attn.k_proj.weight', 'text_model.encoder.layers.4.mlp.fc2.weight', 'text_model.encoder.layers.11.self_attn.v_proj.bias', 'text_model.encoder.layers.11.layer_norm1.weight', 'text_model.encoder.layers.1.self_attn.out_proj.bias', 'text_model.encoder.layers.5.self_attn.v_proj.weight', 'text_model.encoder.layers.5.layer_norm2.bias', 'text_model.encoder.layers.2.self_attn.v_proj.bias', 'text_model.encoder.layers.4.layer_norm2.weight', 'text_model.encoder.layers.8.self_attn.v_proj.bias', 'text_model.encoder.layers.10.layer_norm2.weight', 'text_model.encoder.layers.11.self_attn.k_proj.weight', 'text_model.final_layer_norm.weight', 'text_model.encoder.layers.4.layer_norm2.bias', 'text_model.encoder.layers.10.mlp.fc2.weight', 'text_model.encoder.layers.9.self_attn.v_proj.bias', 'text_model.encoder.layers.5.self_attn.out_proj.bias', 'text_model.encoder.layers.7.layer_norm2.bias', 'text_model.encoder.layers.11.self_attn.out_proj.weight', 'text_model.encoder.layers.11.mlp.fc1.bias', 'text_model.encoder.layers.8.layer_norm2.bias', 'text_model.encoder.layers.6.mlp.fc2.bias', 'text_model.encoder.layers.9.mlp.fc2.bias', 'text_model.encoder.layers.5.mlp.fc1.bias', 'text_model.encoder.layers.8.self_attn.k_proj.weight', 'text_model.encoder.layers.5.mlp.fc2.bias', 'text_model.encoder.layers.0.layer_norm1.weight', 'text_model.encoder.layers.3.mlp.fc2.bias', 'text_projection.weight', 'text_model.encoder.layers.9.self_attn.k_proj.bias', 'text_model.encoder.layers.2.layer_norm2.weight', 'text_model.encoder.layers.10.self_attn.k_proj.weight', 'text_model.encoder.layers.1.mlp.fc1.weight', 'text_model.encoder.layers.5.layer_norm1.weight', 'visual_projection.weight', 'text_model.encoder.layers.9.mlp.fc1.weight', 'text_model.encoder.layers.5.self_attn.out_proj.weight', 'text_model.encoder.layers.0.mlp.fc1.weight', 'text_model.encoder.layers.4.self_attn.v_proj.bias', 'text_model.encoder.layers.2.self_attn.q_proj.bias', 'text_model.encoder.layers.9.self_attn.q_proj.weight', 'text_model.encoder.layers.0.self_attn.k_proj.weight', 'text_model.encoder.layers.3.mlp.fc1.bias', 'text_model.encoder.layers.7.self_attn.v_proj.weight', 'text_model.encoder.layers.6.self_attn.v_proj.weight', 'text_model.encoder.layers.1.self_attn.out_proj.weight', 'text_model.encoder.layers.9.self_attn.out_proj.weight', 'text_model.encoder.layers.8.self_attn.q_proj.bias', 'text_model.encoder.layers.3.self_attn.v_proj.bias', 'text_model.encoder.layers.11.layer_norm2.weight', 'text_model.encoder.layers.9.layer_norm2.weight', 'text_model.encoder.layers.3.self_attn.k_proj.bias', 'text_model.encoder.layers.2.self_attn.v_proj.weight', 'text_model.encoder.layers.10.mlp.fc1.weight', 'text_model.encoder.layers.7.self_attn.out_proj.weight', 'text_model.encoder.layers.7.layer_norm1.weight', 'text_model.encoder.layers.2.layer_norm1.weight', 'text_model.encoder.layers.10.self_attn.out_proj.weight', 'text_model.encoder.layers.10.self_attn.q_proj.weight', 'text_model.encoder.layers.11.mlp.fc2.bias', 'text_model.encoder.layers.9.mlp.fc1.bias', 'text_model.encoder.layers.5.self_attn.k_proj.bias', 'text_model.encoder.layers.11.layer_norm2.bias', 'text_model.encoder.layers.8.self_attn.v_proj.weight', 'text_model.encoder.layers.1.layer_norm2.bias', 'text_model.encoder.layers.1.mlp.fc2.bias', 'text_model.encoder.layers.1.self_attn.k_proj.weight', 'text_model.encoder.layers.9.mlp.fc2.weight', 'text_model.embeddings.position_embedding.weight', 'text_model.encoder.layers.4.self_attn.q_proj.bias', 'text_model.encoder.layers.4.self_attn.out_proj.weight', 'text_model.encoder.layers.5.self_attn.k_proj.weight', 'text_model.encoder.layers.8.layer_norm1.bias', 'text_model.encoder.layers.10.layer_norm1.bias', 'text_model.encoder.layers.3.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.q_proj.bias', 'logit_scale', 'text_model.encoder.layers.8.self_attn.out_proj.weight', 'text_model.encoder.layers.1.layer_norm1.weight', 'text_model.encoder.layers.6.self_attn.v_proj.bias', 'text_model.encoder.layers.6.self_attn.out_proj.bias', 'text_model.encoder.layers.6.layer_norm2.bias', 'text_model.encoder.layers.5.layer_norm1.bias', 'text_model.encoder.layers.3.layer_norm1.bias', 'text_model.encoder.layers.5.layer_norm2.weight', 'text_model.encoder.layers.10.self_attn.v_proj.weight', 'text_model.encoder.layers.2.layer_norm1.bias', 'text_model.encoder.layers.0.self_attn.v_proj.bias', 'text_model.encoder.layers.4.mlp.fc1.weight', 'text_model.encoder.layers.6.mlp.fc1.weight', 'text_model.encoder.layers.8.mlp.fc1.weight', 'text_model.encoder.layers.6.self_attn.k_proj.bias', 'text_model.encoder.layers.7.self_attn.v_proj.bias', 'text_model.encoder.layers.11.layer_norm1.bias', 'text_model.final_layer_norm.bias', 'text_model.encoder.layers.1.mlp.fc2.weight', 'text_model.encoder.layers.0.self_attn.k_proj.bias', 'text_model.encoder.layers.1.mlp.fc1.bias', 'text_model.encoder.layers.2.mlp.fc2.bias', 'text_model.encoder.layers.9.self_attn.v_proj.weight', 'text_model.embeddings.token_embedding.weight', 'text_model.encoder.layers.2.mlp.fc1.weight', 'text_model.encoder.layers.2.self_attn.out_proj.weight', 'text_model.encoder.layers.10.self_attn.q_proj.bias', 'text_model.encoder.layers.5.mlp.fc1.weight', 'text_model.encoder.layers.4.layer_norm1.bias', 'text_model.encoder.layers.3.layer_norm2.bias', 'text_model.encoder.layers.6.mlp.fc2.weight', 'text_model.encoder.layers.0.layer_norm2.weight', 'text_model.encoder.layers.0.self_attn.out_proj.bias', 'text_model.encoder.layers.4.self_attn.out_proj.bias', 'text_model.encoder.layers.2.self_attn.out_proj.bias', 'text_model.encoder.layers.3.self_attn.out_proj.weight', 'text_model.encoder.layers.4.self_attn.k_proj.bias', 'text_model.encoder.layers.11.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.out_proj.weight', 'text_model.encoder.layers.5.self_attn.q_proj.bias', 'text_model.encoder.layers.8.mlp.fc2.bias', 'text_model.encoder.layers.11.self_attn.k_proj.bias', 'text_model.encoder.layers.7.mlp.fc1.weight', 'text_model.encoder.layers.2.mlp.fc2.weight', 'text_model.encoder.layers.0.mlp.fc2.bias', 'text_model.encoder.layers.6.mlp.fc1.bias', 'text_model.encoder.layers.1.layer_norm2.weight', 'text_model.encoder.layers.3.self_attn.k_proj.weight', 'text_model.encoder.layers.0.mlp.fc2.weight', 'text_model.encoder.layers.6.layer_norm1.weight', 'text_model.encoder.layers.6.layer_norm1.bias', 'text_model.encoder.layers.11.self_attn.q_proj.bias', 'text_model.encoder.layers.2.layer_norm2.bias', 'text_model.encoder.layers.8.layer_norm1.weight', 'text_model.encoder.layers.7.layer_norm2.weight', 'text_model.encoder.layers.7.mlp.fc2.weight', 'text_model.encoder.layers.11.mlp.fc2.weight', 'text_model.encoder.layers.10.self_attn.k_proj.bias', 'text_model.encoder.layers.9.self_attn.out_proj.bias', 'text_model.encoder.layers.8.mlp.fc2.weight', 'text_model.encoder.layers.3.layer_norm1.weight', 'text_model.encoder.layers.3.mlp.fc1.weight', 'text_model.encoder.layers.10.mlp.fc2.bias', 'text_model.encoder.layers.6.self_attn.q_proj.weight', 'text_model.encoder.layers.9.layer_norm1.weight', 'text_model.encoder.layers.3.self_attn.out_proj.bias', 'text_model.encoder.layers.0.layer_norm2.bias', 'text_model.encoder.layers.0.self_attn.q_proj.bias', 'text_model.encoder.layers.9.layer_norm1.bias', 'text_model.encoder.layers.11.self_attn.v_proj.weight', 'text_model.encoder.layers.0.self_attn.q_proj.weight', 'text_model.encoder.layers.2.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.k_proj.weight', 'text_model.encoder.layers.5.self_attn.q_proj.weight', 'text_model.encoder.layers.7.layer_norm1.bias', 'text_model.encoder.layers.10.mlp.fc1.bias', 'text_model.encoder.layers.7.self_attn.out_proj.bias', 'text_model.encoder.layers.10.layer_norm1.weight', 'text_model.encoder.layers.10.layer_norm2.bias', 'text_model.encoder.layers.8.self_attn.k_proj.bias', 'text_model.encoder.layers.1.self_attn.v_proj.weight', 'text_model.encoder.layers.3.layer_norm2.weight', 'text_model.embeddings.position_ids', 'text_model.encoder.layers.7.self_attn.q_proj.bias', 'text_model.encoder.layers.4.self_attn.q_proj.weight', 'text_model.encoder.layers.1.self_attn.k_proj.bias', 'text_model.encoder.layers.4.mlp.fc1.bias', 'text_model.encoder.layers.1.self_attn.v_proj.bias', 'text_model.encoder.layers.3.mlp.fc2.weight', 'text_model.encoder.layers.4.self_attn.v_proj.weight', 'text_model.encoder.layers.0.mlp.fc1.bias', 'text_model.encoder.layers.7.self_attn.k_proj.bias', 'text_model.encoder.layers.8.self_attn.q_proj.weight', 'text_model.encoder.layers.0.layer_norm1.bias', 'text_model.encoder.layers.11.mlp.fc1.weight', 'text_model.encoder.layers.0.self_attn.v_proj.weight', 'text_model.encoder.layers.11.self_attn.out_proj.bias', 'text_model.encoder.layers.4.mlp.fc2.bias', 'text_model.encoder.layers.10.self_attn.out_proj.bias', 'text_model.encoder.layers.7.self_attn.k_proj.weight', 'text_model.encoder.layers.2.self_attn.k_proj.weight', 'text_model.encoder.layers.1.layer_norm1.bias', 'text_model.encoder.layers.8.mlp.fc1.bias', 'text_model.encoder.layers.8.layer_norm2.weight', 'text_model.encoder.layers.3.self_attn.v_proj.weight', 'text_model.encoder.layers.0.self_attn.out_proj.weight', 'text_model.encoder.layers.2.mlp.fc1.bias', 'text_model.encoder.layers.10.self_attn.v_proj.bias', 'text_model.encoder.layers.5.self_attn.v_proj.bias', 'text_model.encoder.layers.1.self_attn.q_proj.weight', 'text_model.encoder.layers.7.mlp.fc1.bias', 'text_model.encoder.layers.9.layer_norm2.bias', 'text_model.encoder.layers.2.self_attn.k_proj.bias', 'text_model.encoder.layers.9.self_attn.q_proj.bias', 'text_model.encoder.layers.4.layer_norm1.weight', 'text_model.encoder.layers.9.self_attn.k_proj.weight', 'text_model.encoder.layers.7.self_attn.q_proj.weight', 'text_model.encoder.layers.7.mlp.fc2.bias', 'text_model.encoder.layers.3.self_attn.q_proj.bias', 'text_model.encoder.layers.1.self_attn.q_proj.bias', 'text_model.encoder.layers.8.self_attn.out_proj.bias', 'text_model.encoder.layers.6.layer_norm2.weight', 'text_model.encoder.layers.5.mlp.fc2.weight'] - This IS expected if you are initializing CLIPVisionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing CLIPVisionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). [2024-07-24 19:46:20,610] Number of newly added tokens: 7 [2024-07-24 19:46:20,611] Pre-training stage [2024-07-24 19:46:20,615] BUILDING PROCESSOR 1: clip_image Using /home/zhangzy/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /home/zhangzy/.cache/torch_extensions/py38_cu117/cpu_adam/build.ninja... Building extension module cpu_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module cpu_adam... Time to load cpu_adam op: 3.719231367111206 seconds Using /home/zhangzy/.cache/torch_extensions/py38_cu117 as PyTorch extensions root... Emitting ninja build file /home/zhangzy/.cache/torch_extensions/py38_cu117/utils/build.ninja... Building extension module utils... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module utils... Time to load utils op: 0.8467190265655518 seconds [1] 156387 segmentation fault (core dumped) python train_ullava_core.py --cfg_path ./configs/train/ullava_core_stage1.yam
Hello, please add my wechat: jinxu_95 Ps. I will update the solution after discussion.
solved, deepspeed install error. One can disable deepspeed for debugging.
First thx for Greate work.
How to solve the bug segmentation fault (core dumped) ? The shell output is above.