vision model layer not found in the base model during merging

[NandhaKishorM]

when I run

!bash scripts/merge_lora.sh
after Loading LLama3.2-Vision from base model... Loading LoRA weights... Traceback (most recent call last):

raise ValueError( ValueError: Target modules {'vision_model.transformer.layers.19.self_attn.q_proj', 'vision_model.transformer.layers.30.mlp.fc2', 'language_model.model.layers.15.self_attn.k_proj', 'vision_model.global_transformer.layers.7.self_attn.v_proj', 'language_model.model.layers.32.mlp.up_proj', 'language_model.model.layers.8.mlp.down_proj', 'vision_model.transformer.layers.1.mlp.fc2', 'vision_model.transformer.layers.4.self_attn.q_proj', 'language_model.model.layers.14.self_attn.k_proj', 'vision_model.transformer.layers.11.self_attn.v_proj', 'language_model.model.layers.29.mlp.up_proj', 'language_model.model.layers.21.self_attn.o_proj', 'vision_model.transformer.layers.19.mlp.fc1', 'language_model.model.layers.3.mlp.gate_proj', 'vision_model.transformer.layers.29.mlp.fc1', 'language_model.model.layers.26.self_attn.v_proj', 'language_model.model.layers.35.mlp.up_proj', 'language_model.model.layers.35.mlp.down_proj', 'language_model.model.layers.37.mlp.gate_proj', 'vision_model.transformer.layers.20.self_attn.o_proj', 'language_model.model.layers.8.mlp.up_proj', 'language_model.model.layers.15.mlp.gate_proj', 'language_model.model.layers.7.self_attn.v_proj', 'language_model.model.layers.27.self_attn.v_proj', 'language_model.model.layers.2.self_attn.q_proj', 'vision_model.transformer.layers.21.self_attn.o_proj', 'language_model.model.layers.4.self_attn.q_proj', 'vision_model.transformer.layers.21.self_attn.k_proj', 'language_model.model.layers.36.self_attn.q_proj', 'language_model.model.layers.26.mlp.up_proj', 'vision_model.transformer.layers.27.self_attn.q_proj', 'vision_model.transformer.layers.18.self_attn.k_proj', 'language_model.model.layers.18.cross_attn.q_proj', 'language_model.model.layers.25.mlp.gate_proj', 'language_model.model.layers.29.self_attn.q_proj', 'vision_model.global_transformer.layers.5.mlp.fc1', 'vision_model.transformer.layers.20.mlp.fc2', 'vision_model.transformer.layers.10.self_attn.o_proj', 'language_model.model.layers.29.self_attn.v_proj', 'vision_model.transformer.layers.12.self_attn.o_proj', 'language_model.model.layers.10.mlp.down_proj', 'vision_model.transformer.layers.8.mlp.fc1', 'language_model.model.layers.14.self_attn.v_proj', 'language_model.model.layers.38.mlp.down_proj', 'vision_model.transformer.layers.26.self_attn.q_proj', 'language_model.model.layers.25.self_attn.v_proj', 'vision_model.global_transformer.layers.2.self_attn.o_proj', 'vision_model.transformer.layers.20.self_attn.q_proj', 'vision_model.transformer.layers.21.self_attn.q_proj', 'vision_model.transformer.layers.30.self_attn.q_proj', 'language_model.model.layers.0.self_attn.q_proj', 'language_model.model.layers.39.self_attn.o_proj', 'language_model.model.layers.5.mlp.gate_proj', 'language_model.model.layers.28.cross_attn.v_proj', 'vision_model.transformer.layers.18.mlp.fc2', 'vision_model.transformer.layers.9.mlp.fc1', 'language_model.model.layers.24.self_attn.v_proj', 'language_model.model.layers.28.mlp.up_proj', 'language_model.model.layers.30.self_attn.q_proj', 'vision_model.global_transformer.layers.2.mlp.fc1', 'language_model.model.layers.13.cross_attn.o_proj', 'vision_model.transformer.layers.10.self_attn.v_proj', 'vision_model.global_transformer.layers.1.self_attn.q_proj', 'language_model.model.layers.18.mlp.up_proj', 'vision_model.transformer.layers.26.mlp.fc2', 'language_model.model.layers.28.cross_attn.q_proj', 'vision_model.global_transformer.layers.6.self_attn.k_proj', 'language_model.model.layers.10.self_attn.v_proj', 'language_model.model.layers.2.mlp.gate_proj', 'language_model.model.layers.38.cross_attn.q_proj', 'language_model.model.layers.3.cross_attn.o_proj', 'vision_model.transformer.layers.22.mlp.fc2', 'language_model.model.layers.19.self_attn.o_proj', 'vision_model.transformer.layers.4.mlp.fc1', 'vision_model.transformer.layers.11.mlp.fc1', 'vision_model.transformer.layers.7.self_attn.k_proj', 'vision_model.global_transformer.layers.4.self_attn.k_proj', 'language_model.model.layers.1.mlp.up_proj', 'language_model.model.layers.36.mlp.down_proj', 'language_model.model.layers.38.mlp.up_proj', 'vision_model.transformer.layers.0.self_attn.o_proj', 'vision_model.transformer.layers.14.self_attn.v_proj', 'language_model.model.layers.34.self_attn.o_proj', 'vision_model.transformer.layers.25.self_attn.o_proj', 'language_model.model.layers.34.self_attn.v_proj', 'vision_model.transformer.layers.27.mlp.fc2', 'language_model.model.layers.16.self_attn.o_proj', 'vision_model.transformer.layers.12.mlp.fc1', 'vision_model.transformer.layers.8.self_attn.k_proj', 'vision_model.transformer.layers.17.self_attn.k_proj', 'vision_model.transformer.layers.16.self_attn.v_proj', 'vision_model.transformer.layers.0.mlp.fc2', 'language_model.model.layers.33.cross_attn.q_proj', 'language_model.model.layers.27.mlp.down_proj', 'language_model.model.layers.35.self_attn.v_proj', 'vision_model.transformer.layers.22.self_attn.k_proj', 'vision_model.transformer.layers.3.self_attn.o_proj', 'language_model.model.layers.29.self_attn.o_proj', 'vision_model.transformer.layers.27.self_attn.v_proj', 'language_model.model.layers.12.self_attn.k_proj', 'vision_model.global_transformer.layers.5.self_attn.k_proj', 'vision_model.transformer.layers.26.mlp.fc1', 'language_model.model.layers.29.self_attn.k_proj', 'vision_model.transformer.layers.28.self_attn.q_proj', 'language_model.model.layers.4.mlp.gate_proj', 'vision_model.transformer.layers.2.mlp.fc2', 'language_model.model.layers.1.self_attn.v_proj', 'vision_model.global_transformer.layers.1.self_attn.k_proj', 'language_model.model.layers.28.mlp.gate_proj', 'language_model.model.layers.7.self_attn.o_proj', 'vision_model.transformer.layers.25.mlp.fc1', 'vision_model.transformer.layers.23.mlp.fc1', 'vision_model.transformer.layers.12.self_attn.q_proj', 'language_model.model.layers.30.self_attn.v_proj', 'vision_model.transformer.layers.15.mlp.fc2', 'language_model.model.layers.14.mlp.gate_proj', 'language_model.model.layers.31.mlp.gate_proj', 'language_model.model.layers.1.mlp.gate_proj', 'language_model.model.layers.11.mlp.up_proj', 'vision_model.transformer.layers.3.self_attn.v_proj', 'language_model.model.layers.13.mlp.gate_proj', 'language_model.model.layers.7.mlp.down_proj', 'vision_model.transformer.layers.3.self_attn.q_proj', 'vision_model.transformer.layers.5.self_attn.q_proj', 'language_model.model.layers.11.self_attn.v_proj', 'language_model.model.layers.1.self_attn.q_proj', 'vision_model.transformer.layers.20.mlp.fc1', 'vision_model.global_transformer.layers.3.self_attn.o_proj', 'vision_model.transformer.layers.31.self_attn.q_proj', 'vision_model.transformer.layers.13.self_attn.k_proj', 'language_model.model.layers.19.mlp.gate_proj', 'language_model.model.layers.24.mlp.up_proj', 'vision_model.transformer.layers.13.self_attn.q_proj', 'language_model.model.layers.34.mlp.up_proj', 'vision_model.transformer.layers.8.mlp.fc2', 'vision_model.transformer.layers.21.self_attn.v_proj', 'vision_model.transformer.layers.23.self_attn.k_proj', 'language_model.model.layers.23.mlp.down_proj', 'language_model.model.layers.28.cross_attn.k_proj', 'vision_model.transformer.layers.30.self_attn.o_proj', 'language_model.model.layers.30.self_attn.k_proj', 'language_model.model.layers.27.self_attn.k_proj', 'language_model.model.layers.9.self_attn.o_proj', 'vision_model.global_transformer.layers.4.self_attn.v_proj', 'vision_model.global_transformer.layers.0.mlp.fc1', 'language_model.model.layers.22.self_attn.v_proj', 'vision_model.transformer.layers.18.self_attn.v_proj', 'vision_model.transformer.layers.20.self_attn.v_proj', 'language_model.model.layers.7.mlp.up_proj', 'language_model.model.layers.12.mlp.gate_proj', 'vision_model.transformer.layers.29.mlp.fc2', 'vision_model.transformer.layers.30.mlp.fc1', 'language_model.model.layers.27.self_attn.q_proj', 'language_model.model.layers.2.self_attn.o_proj', 'vision_model.global_transformer.layers.6.self_attn.v_proj', 'language_model.model.layers.31.self_attn.q_proj', 'vision_model.global_transformer.layers.1.self_attn.o_proj', 'language_model.model.layers.19.mlp.down_proj', 'vision_model.transformer.layers.4.mlp.fc2', 'vision_model.transformer.layers.17.mlp.fc1', 'language_model.model.layers.19.self_attn.q_proj', 'language_model.model.layers.15.mlp.up_proj', 'language_model.model.layers.21.self_attn.v_proj', 'vision_model.global_transformer.layers.4.mlp.fc1', 'language_model.model.layers.8.cross_attn.v_proj', 'vision_model.global_transformer.layers.5.mlp.fc2', 'language_model.model.layers.31.self_attn.k_proj', 'vision_model.transformer.layers.12.self_attn.k_proj', 'vision_model.transformer.layers.23.mlp.fc2', 'vision_model.transformer.layers.28.self_attn.v_proj', 'language_model.model.layers.25.mlp.up_proj', 'vision_model.transformer.layers.24.self_attn.v_proj', 'vision_model.transformer.layers.15.self_attn.v_proj', 'vision_model.transformer.layers.24.mlp.fc2', 'vision_model.transformer.layers.19.mlp.fc2', 'language_model.model.layers.29.mlp.down_proj', 'vision_model.transformer.layers.4.self_attn.o_proj', 'language_model.model.layers.32.self_attn.v_proj', 'vision_model.global_transformer.layers.0.self_attn.q_proj', 'language_model.model.layers.20.mlp.up_proj', 'language_model.model.layers.0.mlp.up_proj', 'language_model.model.layers.26.self_attn.k_proj', 'language_model.model.layers.9.self_attn.v_proj', 'vision_model.global_transformer.layers.3.self_attn.k_proj', 'language_model.model.layers.8.mlp.gate_proj', 'vision_model.transformer.layers.0.mlp.fc1', 'vision_model.transformer.layers.24.self_attn.o_proj', 'vision_model.global_transformer.layers.1.mlp.fc2', 'language_model.model.layers.10.self_attn.q_proj', 'language_model.model.layers.18.cross_attn.v_proj', 'vision_model.transformer.layers.6.self_attn.k_proj', 'language_model.model.layers.21.self_attn.q_proj', 'language_model.model.layers.26.self_attn.q_proj', 'vision_model.global_transformer.layers.1.self_attn.v_proj', 'language_model.model.layers.18.cross_attn.k_proj', 'vision_model.transformer.layers.31.self_attn.v_proj', 'vision_model.global_transformer.layers.6.mlp.fc1', 'vision_model.transformer.layers.9.self_attn.v_proj', 'language_model.model.layers.30.self_attn.o_proj', 'vision_model.transformer.layers.30.self_attn.v_proj', 'vision_model.transformer.layers.12.mlp.fc2', 'vision_model.post_tile_positional_embedding.embedding', 'language_model.model.layers.3.cross_attn.v_proj', 'vision_model.global_transformer.layers.2.self_attn.k_proj', 'language_model.model.layers.23.cross_attn.q_proj', 'language_model.model.layers.5.mlp.down_proj', 'vision_model.transformer.layers.10.mlp.fc1', 'vision_model.transformer.layers.13.self_attn.o_proj', 'vision_model.transformer.layers.26.self_attn.o_proj', 'vision_model.transformer.layers.3.mlp.fc2', 'vision_model.transformer.layers.15.mlp.fc1', 'language_model.model.layers.1.self_attn.k_proj', 'language_model.model.layers.36.self_attn.o_proj', 'language_model.model.layers.24.mlp.down_proj', 'vision_model.transformer.layers.31.self_attn.k_proj', 'language_model.model.layers.4.mlp.up_proj', 'vision_model.global_transformer.layers.5.self_attn.v_proj', 'language_model.model.layers.22.mlp.gate_proj', 'language_model.model.layers.7.self_attn.k_proj', 'language_model.model.layers.26.self_attn.o_proj', 'language_model.model.layers.3.cross_attn.k_proj', 'vision_model.global_transformer.layers.4.self_attn.o_proj', 'language_model.model.layers.7.self_attn.q_proj', 'language_model.model.layers.35.self_attn.k_proj', 'language_model.model.layers.25.self_attn.q_proj', 'vision_model.global_transformer.layers.2.mlp.fc2', 'vision_model.transformer.layers.6.self_attn.v_proj', 'vision_model.transformer.layers.25.self_attn.q_proj', 'vision_model.transformer.layers.1.self_attn.v_proj', 'vision_model.global_transformer.layers.1.mlp.fc1', 'vision_model.transformer.layers.16.mlp.fc2', 'language_model.model.layers.9.self_attn.k_proj', 'language_model.model.layers.10.mlp.gate_proj', 'language_model.model.layers.30.mlp.up_proj', 'language_model.model.layers.3.cross_attn.q_proj', 'vision_model.global_transformer.layers.7.mlp.fc2', 'vision_model.transformer.layers.14.mlp.fc1', 'language_model.model.layers.24.self_attn.o_proj', 'language_model.model.layers.17.self_attn.k_proj', 'vision_model.transformer.layers.1.mlp.fc1', 'vision_model.global_transformer.layers.7.self_attn.o_proj', 'language_model.model.layers.39.self_attn.k_proj', 'language_model.model.layers.21.mlp.up_proj', 'vision_model.transformer.layers.15.self_attn.k_proj', 'language_model.model.layers.23.cross_attn.v_proj', 'language_model.model.layers.1.self_attn.o_proj', 'language_model.model.layers.8.cross_attn.k_proj', 'language_model.model.layers.37.self_attn.v_proj', 'vision_model.transformer.layers.5.self_attn.k_proj', 'language_model.model.layers.39.mlp.up_proj', 'language_model.model.layers.17.self_attn.v_proj', 'language_model.model.layers.15.self_attn.q_proj', 'language_model.model.layers.18.mlp.down_proj', 'language_model.model.layers.14.mlp.down_proj', 'language_model.model.layers.31.self_attn.v_proj', 'vision_model.transformer.layers.20.self_attn.k_proj', 'language_model.model.layers.22.self_attn.k_proj', 'language_model.model.layers.26.mlp.down_proj', 'language_model.model.layers.15.self_attn.o_proj', 'language_model.model.layers.12.mlp.down_proj', 'vision_model.transformer.layers.19.self_attn.v_proj', 'language_model.model.layers.20.mlp.gate_proj', 'language_model.model.layers.0.mlp.gate_proj', 'vision_model.transformer.layers.22.self_attn.o_proj', 'language_model.model.layers.23.cross_attn.o_proj', 'language_model.model.layers.8.cross_attn.o_proj', 'language_model.model.layers.17.self_attn.q_proj', 'multi_modal_projector', 'vision_model.transformer.layers.31.self_attn.o_proj', 'language_model.model.layers.37.mlp.down_proj', 'language_model.model.layers.36.self_attn.v_proj', 'language_model.model.layers.25.mlp.down_proj', 'language_model.model.layers.17.mlp.up_proj', 'vision_model.transformer.layers.29.self_attn.q_proj', 'language_model.model.layers.27.mlp.gate_proj', 'vision_model.transformer.layers.14.self_attn.k_proj', 'language_model.model.layers.24.mlp.gate_proj', 'language_model.model.layers.16.self_attn.k_proj', 'vision_model.transformer.layers.16.self_attn.k_proj', 'language_model.model.layers.21.mlp.gate_proj', 'vision_model.transformer.layers.5.self_attn.v_proj', 'language_model.model.layers.6.mlp.gate_proj', 'language_model.model.layers.21.mlp.down_proj', 'language_model.model.layers.16.mlp.up_proj', 'vision_model.transformer.layers.3.mlp.fc1', 'vision_model.transformer.layers.7.self_attn.q_proj', 'vision_model.transformer.layers.17.mlp.fc2', 'language_model.model.layers.29.mlp.gate_proj', 'language_model.model.layers.13.cross_attn.k_proj', 'language_model.model.layers.14.mlp.up_proj', 'language_model.model.layers.5.self_attn.v_proj', 'language_model.model.layers.2.mlp.down_proj', 'vision_model.transformer.layers.31.mlp.fc1', 'language_model.model.layers.22.mlp.down_proj', 'language_model.model.layers.32.mlp.down_proj', 'language_model.model.layers.31.mlp.down_proj', 'vision_model.global_transformer.layers.6.self_attn.q_proj', 'language_model.model.layers.39.mlp.gate_proj', 'vision_model.transformer.layers.24.self_attn.k_proj', 'language_model.model.layers.4.self_attn.k_proj', 'language_model.model.layers.37.self_attn.k_proj', 'language_model.model.layers.12.self_attn.o_proj', 'language_model.model.layers.3.mlp.up_proj', 'vision_model.transformer.layers.6.self_attn.q_proj', 'vision_model.transformer.layers.29.self_attn.o_proj', 'vision_model.transformer.layers.5.mlp.fc2', 'language_model.model.layers.26.mlp.gate_proj', 'vision_model.transformer.layers.28.self_attn.k_proj', 'vision_model.transformer.layers.27.self_attn.o_proj', 'vision_model.transformer.layers.0.self_attn.v_proj', 'language_model.model.layers.10.self_attn.k_proj', 'vision_model.transformer.layers.17.self_attn.v_proj', 'vision_model.transformer.layers.10.mlp.fc2', 'language_model.model.layers.11.self_attn.q_proj', 'vision_model.gated_positional_embedding.tile_embedding', 'language_model.model.layers.31.self_attn.o_proj', 'vision_model.global_transformer.layers.3.mlp.fc1', 'vision_model.global_transformer.layers.7.self_attn.k_proj', 'language_model.model.layers.23.mlp.gate_proj', 'language_model.model.layers.21.self_attn.k_proj', 'vision_model.global_transformer.layers.0.mlp.fc2', 'language_model.model.layers.16.mlp.down_proj', 'language_model.model.layers.6.self_attn.k_proj', 'vision_model.transformer.layers.27.mlp.fc1', 'language_model.model.layers.5.self_attn.q_proj', 'language_model.model.layers.16.mlp.gate_proj', 'vision_model.transformer.layers.26.self_attn.k_proj', 'vision_model.transformer.layers.30.self_attn.k_proj', 'language_model.model.layers.14.self_attn.o_proj', 'vision_model.transformer.layers.6.mlp.fc2', 'language_model.model.layers.15.self_attn.v_proj', 'vision_model.transformer.layers.2.self_attn.q_proj', 'language_model.model.layers.31.mlp.up_proj', 'vision_model.transformer.layers.16.mlp.fc1', 'vision_model.transformer.layers.29.self_attn.k_proj', 'vision_model.transformer.layers.11.self_attn.o_proj', 'vision_model.transformer.layers.7.self_attn.v_proj', 'language_model.model.layers.2.mlp.up_proj', 'language_model.model.layers.19.self_attn.v_proj', 'language_model.model.layers.36.mlp.up_proj', 'language_model.model.layers.6.mlp.up_proj', 'vision_model.transformer.layers.8.self_attn.q_proj', 'language_model.model.layers.24.self_attn.k_proj', 'vision_model.transformer.layers.14.mlp.fc2', 'vision_model.transformer.layers.31.mlp.fc2', 'vision_model.global_transformer.layers.6.self_attn.o_proj', 'vision_model.transformer.layers.25.self_attn.v_proj', 'language_model.model.layers.1.mlp.down_proj', 'language_model.model.layers.34.mlp.gate_proj', 'language_model.model.layers.38.cross_attn.o_proj', 'language_model.model.layers.9.self_attn.q_proj', 'language_model.model.layers.9.mlp.down_proj', 'language_model.model.layers.13.mlp.up_proj', 'vision_model.transformer.layers.9.self_attn.o_proj', 'language_model.model.layers.18.cross_attn.o_proj', 'language_model.model.layers.10.mlp.up_proj', 'vision_model.transformer.layers.5.self_attn.o_proj', 'language_model.model.layers.8.cross_attn.q_proj', 'vision_model.transformer.layers.21.mlp.fc1', 'vision_model.transformer.layers.7.mlp.fc2', 'vision_model.transformer.layers.19.self_attn.o_proj', 'vision_model.transformer.layers.15.self_attn.o_proj', 'vision_model.transformer.layers.3.self_attn.k_proj', 'vision_model.transformer.layers.16.self_attn.q_proj', 'vision_model.transformer.layers.11.mlp.fc2', 'language_model.model.layers.19.self_attn.k_proj', 'language_model.model.layers.20.mlp.down_proj', 'language_model.model.layers.12.mlp.up_proj', 'vision_model.global_transformer.layers.2.self_attn.v_proj', 'language_model.model.layers.11.self_attn.o_proj', 'vision_model.transformer.layers.12.self_attn.v_proj', 'language_model.model.layers.12.self_attn.v_proj', 'vision_model.transformer.layers.22.mlp.fc1', 'language_model.model.layers.6.mlp.down_proj', 'language_model.model.layers.25.self_attn.o_proj', 'language_model.model.layers.23.mlp.up_proj', 'language_model.model.layers.34.self_attn.q_proj', 'language_model.model.layers.36.self_attn.k_proj', 'language_model.model.layers.33.mlp.up_proj', 'language_model.model.layers.25.self_attn.k_proj', 'vision_model.transformer.layers.24.mlp.fc1', 'vision_model.transformer.layers.1.self_attn.q_proj', 'language_model.model.layers.22.self_attn.o_proj', 'language_model.model.layers.6.self_attn.q_proj', 'vision_model.transformer.layers.14.self_attn.o_proj', 'vision_model.transformer.layers.0.self_attn.q_proj', 'vision_model.transformer.layers.19.self_attn.k_proj', 'language_model.model.layers.17.mlp.gate_proj', 'language_model.model.layers.34.self_attn.k_proj', 'vision_model.transformer.layers.9.self_attn.k_proj', 'language_model.model.layers.30.mlp.gate_proj', 'language_model.model.layers.28.cross_attn.o_proj', 'language_model.model.layers.16.self_attn.q_proj', 'vision_model.transformer.layers.1.self_attn.o_proj', 'language_model.model.layers.9.mlp.gate_proj', 'vision_model.transformer.layers.10.self_attn.k_proj', 'language_model.model.layers.11.self_attn.k_proj', 'language_model.model.layers.16.self_attn.v_proj', 'vision_model.transformer.layers.28.mlp.fc1', 'language_model.model.layers.20.self_attn.k_proj', 'vision_model.transformer.layers.2.self_attn.k_proj', 'language_model.model.layers.9.mlp.up_proj', 'language_model.model.layers.33.cross_attn.v_proj', 'language_model.model.layers.17.mlp.down_proj', 'vision_model.global_transformer.layers.5.self_attn.o_proj', 'vision_model.transformer.layers.13.mlp.fc1', 'language_model.model.layers.34.mlp.down_proj', 'vision_model.transformer.layers.13.self_attn.v_proj', 'vision_model.transformer.layers.8.self_attn.v_proj', 'language_model.model.layers.32.self_attn.o_proj', 'vision_model.transformer.layers.15.self_attn.q_proj', 'vision_model.transformer.layers.16.self_attn.o_proj', 'vision_model.transformer.layers.9.self_attn.q_proj', 'language_model.model.layers.0.self_attn.k_proj', 'vision_model.transformer.layers.22.self_attn.q_proj', 'language_model.model.layers.33.cross_attn.k_proj', 'language_model.model.layers.15.mlp.down_proj', 'vision_model.transformer.layers.8.self_attn.o_proj', 'language_model.model.layers.37.self_attn.q_proj', 'language_model.model.layers.28.mlp.down_proj', 'language_model.model.layers.39.self_attn.v_proj', 'vision_model.transformer.layers.27.self_attn.k_proj', 'language_model.model.layers.0.self_attn.v_proj', 'vision_model.transformer.layers.25.mlp.fc2', 'vision_model.transformer.layers.0.self_attn.k_proj', 'language_model.model.layers.2.self_attn.k_proj', 'vision_model.transformer.layers.5.mlp.fc1', 'vision_model.transformer.layers.18.mlp.fc1', 'language_model.model.layers.27.self_attn.o_proj', 'language_model.model.layers.37.self_attn.o_proj', 'vision_model.transformer.layers.9.mlp.fc2', 'language_model.model.layers.3.mlp.down_proj', 'vision_model.transformer.layers.2.self_attn.v_proj', 'vision_model.global_transformer.layers.3.self_attn.q_proj', 'language_model.model.layers.30.mlp.down_proj', 'vision_model.transformer.layers.7.self_attn.o_proj', 'language_model.model.layers.11.mlp.down_proj', 'vision_model.transformer.layers.24.self_attn.q_proj', 'language_model.model.layers.4.self_attn.o_proj', 'language_model.model.layers.10.self_attn.o_proj', 'vision_model.transformer.layers.17.self_attn.o_proj', 'vision_model.transformer.layers.28.mlp.fc2', 'vision_model.transformer.layers.22.self_attn.v_proj', 'language_model.model.layers.11.mlp.gate_proj', 'vision_model.transformer.layers.4.self_attn.v_proj', 'vision_model.transformer.layers.6.mlp.fc1', 'vision_model.global_transformer.layers.3.self_attn.v_proj', 'vision_model.transformer.layers.10.self_attn.q_proj', 'language_model.model.layers.0.self_attn.o_proj', 'language_model.model.layers.2.self_attn.v_proj', 'vision_model.transformer.layers.13.mlp.fc2', 'language_model.model.layers.32.self_attn.k_proj', 'vision_model.transformer.layers.23.self_attn.v_proj', 'vision_model.transformer.layers.26.self_attn.v_proj', 'language_model.model.layers.5.self_attn.o_proj', 'vision_model.transformer.layers.1.self_attn.k_proj', 'language_model.model.layers.32.mlp.gate_proj', 'language_model.model.layers.35.self_attn.o_proj', 'language_model.model.layers.5.self_attn.k_proj', 'vision_model.transformer.layers.6.self_attn.o_proj', 'language_model.model.layers.35.self_attn.q_proj', 'language_model.model.layers.35.mlp.gate_proj', 'language_model.model.layers.7.mlp.gate_proj', 'language_model.model.layers.20.self_attn.o_proj', 'language_model.model.layers.14.self_attn.q_proj', 'language_model.model.layers.33.cross_attn.o_proj', 'language_model.model.layers.19.mlp.up_proj', 'language_model.model.layers.17.self_attn.o_proj', 'language_model.model.layers.20.self_attn.v_proj', 'language_model.model.layers.22.self_attn.q_proj', 'language_model.model.layers.39.mlp.down_proj', 'vision_model.transformer.layers.28.self_attn.o_proj', 'vision_model.global_transformer.layers.4.self_attn.q_proj', 'vision_model.global_transformer.layers.5.self_attn.q_proj', 'vision_model.global_transformer.layers.2.self_attn.q_proj', 'vision_model.transformer.layers.29.self_attn.v_proj', 'language_model.model.layers.6.self_attn.v_proj', 'language_model.model.layers.27.mlp.up_proj', 'language_model.model.layers.13.cross_attn.q_proj', 'language_model.model.layers.18.mlp.gate_proj', 'language_model.model.layers.33.mlp.gate_proj', 'vision_model.transformer.layers.14.self_attn.q_proj', 'vision_model.transformer.layers.2.mlp.fc1', 'language_model.model.layers.36.mlp.gate_proj', 'vision_model.transformer.layers.17.self_attn.q_proj', 'language_model.model.layers.33.mlp.down_proj', 'vision_model.transformer.layers.23.self_attn.q_proj', 'vision_model.global_transformer.layers.0.self_attn.v_proj', 'vision_model.transformer.layers.11.self_attn.k_proj', 'vision_model.transformer.layers.7.mlp.fc1', 'vision_model.global_transformer.layers.6.mlp.fc2', 'vision_model.transformer.layers.21.mlp.fc2', 'language_model.model.layers.38.cross_attn.v_proj', 'vision_model.transformer.layers.4.self_attn.k_proj', 'vision_model.global_transformer.layers.0.self_attn.k_proj', 'language_model.model.layers.24.self_attn.q_proj', 'language_model.model.layers.32.self_attn.q_proj', 'vision_model.pre_tile_positional_embedding.embedding', 'language_model.model.layers.4.self_attn.v_proj', 'language_model.model.layers.0.mlp.down_proj', 'language_model.model.layers.4.mlp.down_proj', 'vision_model.transformer.layers.18.self_attn.o_proj', 'language_model.model.layers.38.mlp.gate_proj', 'vision_model.global_transformer.layers.3.mlp.fc2', 'language_model.model.layers.20.self_attn.q_proj', 'vision_model.transformer.layers.11.self_attn.q_proj', 'language_model.model.layers.6.self_attn.o_proj', 'vision_model.global_transformer.layers.4.mlp.fc2', 'vision_model.transformer.layers.2.self_attn.o_proj', 'language_model.model.layers.23.cross_attn.k_proj', 'language_model.model.layers.39.self_attn.q_proj', 'vision_model.global_transformer.layers.7.self_attn.q_proj', 'vision_model.transformer.layers.23.self_attn.o_proj', 'vision_model.global_transformer.layers.7.mlp.fc1', 'language_model.model.layers.12.self_attn.q_proj', 'language_model.model.layers.38.cross_attn.k_proj', 'language_model.model.layers.13.mlp.down_proj', 'language_model.model.layers.5.mlp.up_proj', 'language_model.model.layers.13.cross_attn.v_proj', 'vision_model.transformer.layers.18.self_attn.q_proj', 'vision_model.transformer.layers.25.self_attn.k_proj', 'language_model.model.layers.22.mlp.up_proj', 'language_model.model.layers.37.mlp.up_proj', 'vision_model.global_transformer.layers.0.self_attn.o_proj'} not found in the base model. Please check the target modules and try again.

[NandhaKishorM]

the problem exists

[NandhaKishorM]

It is happening as I tried fine tuning both vision and llm and try to merge. The base model not merging with the adapter model

[2U1]

Okay if you use LoRA only to LLM that dosen't happen. Is it right?

[2U1]

model = AutoModelForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, trust_remote_code=True, **kwargs) Can you use fix the model loading code in utils.py line 48 like this and try it again?

[NandhaKishorM]

{ "alpha_pattern": {}, "auto_mapping": { "base_model_class": "MllamaForConditionalGeneration", "parent_library": "transformers.models.mllama.modeling_mllama" }, "base_model_name_or_path": "meta-llama/Llama-3.2-11B-Vision-Instruct", "bias": "none", "fan_in_fan_out": false, "inference_mode": true, "init_lora_weights": true, "layer_replication": null, "layers_pattern": null, "layers_to_transform": null, "loftq_config": {}, "lora_alpha": 128, "lora_dropout": 0.05, "megatron_config": null, "megatron_core": "megatron.core", "modules_to_save": null, "peft_type": "LORA", "r": 64, "rank_pattern": {}, "revision": null, "target_modules": [ "language_model.model.layers.28.cross_attn.k_proj", "vision_model.global_transformer.layers.4.self_attn.o_proj", "language_model.model.layers.16.mlp.gate_proj", "language_model.model.layers.25.self_attn.v_proj", "vision_model.transformer.layers.19.mlp.fc1", "language_model.model.layers.28.mlp.gate_proj", "language_model.model.layers.19.mlp.up_proj", "vision_model.transformer.layers.30.mlp.fc2", "language_model.model.layers.33.cross_attn.v_proj", "language_model.model.layers.35.self_attn.k_proj", "language_model.model.layers.34.self_attn.o_proj", "vision_model.transformer.layers.29.self_attn.v_proj", "vision_model.transformer.layers.11.self_attn.k_proj", "vision_model.transformer.layers.2.self_attn.o_proj", "vision_model.transformer.layers.23.mlp.fc1", "language_model.model.layers.16.self_attn.o_proj", "language_model.model.layers.36.self_attn.k_proj", "vision_model.transformer.layers.1.mlp.fc1", "vision_model.transformer.layers.14.self_attn.v_proj", "language_model.model.layers.38.cross_attn.v_proj", "language_model.model.layers.1.mlp.down_proj", "language_model.model.layers.3.cross_attn.o_proj", "language_model.model.layers.15.mlp.up_proj", "vision_model.global_transformer.layers.7.self_attn.v_proj", "language_model.model.layers.36.mlp.up_proj", "language_model.model.layers.11.self_attn.q_proj", "vision_model.transformer.layers.6.self_attn.o_proj", "vision_model.global_transformer.layers.0.self_attn.o_proj", "language_model.model.layers.3.cross_attn.v_proj", "language_model.model.layers.6.self_attn.v_proj", "vision_model.transformer.layers.31.mlp.fc2", "language_model.model.layers.10.self_attn.v_proj", "vision_model.transformer.layers.27.self_attn.o_proj", "language_model.model.layers.16.self_attn.q_proj", "language_model.model.layers.1.mlp.gate_proj", "vision_model.transformer.layers.20.self_attn.k_proj", "vision_model.transformer.layers.16.mlp.fc1", "vision_model.global_transformer.layers.6.self_attn.k_proj", "language_model.model.layers.33.cross_attn.o_proj", "vision_model.transformer.layers.4.self_attn.q_proj", "language_model.model.layers.7.mlp.gate_proj", "language_model.model.layers.26.mlp.down_proj", "language_model.model.layers.35.mlp.gate_proj", "language_model.model.layers.16.mlp.down_proj", "vision_model.transformer.layers.10.self_attn.q_proj", "language_model.model.layers.34.mlp.down_proj", "vision_model.transformer.layers.21.mlp.fc2", "vision_model.transformer.layers.30.self_attn.o_proj", "vision_model.transformer.layers.31.mlp.fc1", "language_model.model.layers.21.mlp.down_proj", "language_model.model.layers.19.mlp.gate_proj", "vision_model.transformer.layers.12.self_attn.o_proj", "language_model.model.layers.2.mlp.down_proj", "language_model.model.layers.8.mlp.gate_proj", "language_model.model.layers.16.self_attn.k_proj", "vision_model.transformer.layers.26.mlp.fc1", "vision_model.global_transformer.layers.5.self_attn.o_proj", "vision_model.transformer.layers.13.mlp.fc1", "vision_model.global_transformer.layers.2.self_attn.o_proj", "vision_model.transformer.layers.16.self_attn.o_proj", "vision_model.global_transformer.layers.6.mlp.fc2", "vision_model.transformer.layers.22.self_attn.q_proj", "language_model.model.layers.26.self_attn.v_proj", "language_model.model.layers.28.cross_attn.q_proj", "language_model.model.layers.5.self_attn.q_proj", "vision_model.global_transformer.layers.2.mlp.fc1", "vision_model.transformer.layers.29.self_attn.k_proj", "language_model.model.layers.29.self_attn.v_proj", "language_model.model.layers.16.self_attn.v_proj", "vision_model.transformer.layers.8.self_attn.o_proj", "language_model.model.layers.13.mlp.up_proj", "language_model.model.layers.20.self_attn.o_proj", "language_model.model.layers.33.mlp.up_proj", "language_model.model.layers.13.cross_attn.v_proj", "vision_model.global_transformer.layers.1.self_attn.v_proj", "vision_model.transformer.layers.14.self_attn.o_proj", "vision_model.transformer.layers.21.self_attn.k_proj", "language_model.model.layers.14.self_attn.q_proj", "vision_model.transformer.layers.0.mlp.fc1", "vision_model.transformer.layers.31.self_attn.q_proj", "language_model.model.layers.37.mlp.down_proj", "language_model.model.layers.18.cross_attn.q_proj", "vision_model.transformer.layers.16.mlp.fc2", "vision_model.post_tile_positional_embedding.embedding", "language_model.model.layers.13.mlp.gate_proj", "language_model.model.layers.27.mlp.up_proj", "language_model.model.layers.3.cross_attn.q_proj", "language_model.model.layers.2.self_attn.q_proj", "language_model.model.layers.37.self_attn.k_proj", "language_model.model.layers.36.self_attn.o_proj", "vision_model.transformer.layers.20.mlp.fc1", "language_model.model.layers.12.mlp.up_proj", "language_model.model.layers.34.mlp.up_proj", "language_model.model.layers.19.self_attn.o_proj", "language_model.model.layers.12.self_attn.o_proj", "vision_model.transformer.layers.31.self_attn.v_proj", "language_model.model.embed_tokens", "language_model.model.layers.30.self_attn.v_proj", "language_model.model.layers.3.cross_attn.k_proj", "vision_model.transformer.layers.28.mlp.fc2", "vision_model.transformer.layers.25.self_attn.q_proj", "vision_model.transformer.layers.2.mlp.fc1", "vision_model.transformer.layers.14.self_attn.q_proj", "vision_model.transformer.layers.4.self_attn.o_proj", "vision_model.global_transformer.layers.5.mlp.fc2", "vision_model.transformer.layers.22.self_attn.v_proj", "vision_model.transformer.layers.28.self_attn.o_proj", "language_model.model.layers.17.self_attn.v_proj", "language_model.model.layers.3.mlp.down_proj", "language_model.model.layers.10.self_attn.q_proj", "language_model.model.layers.35.self_attn.q_proj", "vision_model.transformer.layers.3.mlp.fc1", "vision_model.transformer.layers.20.self_attn.o_proj", "vision_model.transformer.layers.15.self_attn.v_proj", "language_model.model.layers.0.self_attn.k_proj", "language_model.model.layers.7.self_attn.o_proj", "vision_model.transformer.layers.4.mlp.fc1", "vision_model.transformer.layers.9.self_attn.k_proj", "language_model.model.layers.4.self_attn.k_proj", "vision_model.global_transformer.layers.4.self_attn.q_proj", "language_model.model.layers.21.self_attn.o_proj", "language_model.model.layers.8.cross_attn.q_proj", "vision_model.transformer.layers.3.mlp.fc2", "vision_model.transformer.layers.26.self_attn.q_proj", "vision_model.transformer.layers.5.self_attn.q_proj", "language_model.model.layers.15.self_attn.v_proj", "language_model.model.layers.17.self_attn.k_proj", "vision_model.transformer.layers.26.self_attn.k_proj", "vision_model.transformer.layers.7.self_attn.o_proj", "language_model.model.layers.31.mlp.down_proj", "language_model.model.layers.4.mlp.down_proj", "vision_model.transformer.layers.9.mlp.fc1", "language_model.model.layers.9.self_attn.v_proj", "language_model.model.layers.7.self_attn.q_proj", "vision_model.transformer.layers.0.self_attn.o_proj", "vision_model.transformer.layers.26.self_attn.v_proj", "language_model.model.layers.22.mlp.up_proj", "language_model.model.layers.9.mlp.up_proj", "vision_model.transformer.layers.3.self_attn.v_proj", "language_model.model.layers.0.mlp.gate_proj", "language_model.model.layers.28.cross_attn.o_proj", "vision_model.transformer.layers.13.self_attn.o_proj", "vision_model.transformer.layers.0.self_attn.q_proj", "language_model.model.layers.39.mlp.down_proj", "language_model.model.layers.28.mlp.down_proj", "language_model.model.layers.13.cross_attn.q_proj", "vision_model.transformer.layers.24.self_attn.v_proj", "language_model.model.layers.21.self_attn.q_proj", "language_model.model.layers.18.mlp.down_proj", "vision_model.global_transformer.layers.1.mlp.fc2", "vision_model.transformer.layers.13.self_attn.v_proj", "vision_model.global_transformer.layers.7.self_attn.k_proj", "vision_model.transformer.layers.14.mlp.fc2", "vision_model.transformer.layers.1.self_attn.k_proj", "vision_model.transformer.layers.24.self_attn.q_proj", "vision_model.transformer.layers.5.mlp.fc1", "vision_model.transformer.layers.18.self_attn.o_proj", "language_model.model.layers.1.self_attn.v_proj", "language_model.model.layers.15.mlp.down_proj", "language_model.model.layers.23.mlp.down_proj", "language_model.model.layers.7.mlp.down_proj", "vision_model.transformer.layers.11.self_attn.o_proj", "vision_model.transformer.layers.25.self_attn.k_proj", "language_model.model.layers.25.self_attn.q_proj", "language_model.model.layers.23.cross_attn.o_proj", "language_model.model.layers.11.mlp.gate_proj", "language_model.model.layers.11.self_attn.v_proj", "language_model.model.layers.38.cross_attn.q_proj", "vision_model.transformer.layers.29.mlp.fc1", "language_model.model.layers.3.mlp.gate_proj", "language_model.model.layers.1.mlp.up_proj", "vision_model.transformer.layers.22.self_attn.o_proj", "language_model.model.layers.26.self_attn.q_proj", "vision_model.transformer.layers.7.mlp.fc1", "language_model.model.layers.13.cross_attn.o_proj", "vision_model.transformer.layers.10.mlp.fc1", "language_model.model.layers.24.mlp.down_proj", "vision_model.transformer.layers.30.self_attn.v_proj", "language_model.model.layers.20.self_attn.k_proj", "language_model.model.layers.11.self_attn.k_proj", "vision_model.global_transformer.layers.7.mlp.fc2", "language_model.model.layers.5.self_attn.v_proj", "language_model.model.layers.32.self_attn.o_proj", "language_model.model.layers.7.self_attn.k_proj", "vision_model.transformer.layers.31.self_attn.o_proj", "vision_model.transformer.layers.14.self_attn.k_proj", "vision_model.global_transformer.layers.5.self_attn.k_proj", "language_model.model.layers.19.self_attn.v_proj", "language_model.model.layers.39.mlp.gate_proj", "language_model.model.layers.39.self_attn.o_proj", "vision_model.transformer.layers.30.mlp.fc1", "vision_model.transformer.layers.17.self_attn.q_proj", "language_model.model.layers.5.mlp.down_proj", "language_model.model.layers.0.self_attn.o_proj", "vision_model.transformer.layers.0.self_attn.v_proj", "language_model.model.layers.12.self_attn.k_proj", "vision_model.transformer.layers.24.self_attn.k_proj", "vision_model.transformer.layers.5.self_attn.k_proj", "language_model.model.layers.22.self_attn.q_proj", "language_model.model.layers.37.self_attn.o_proj", "vision_model.transformer.layers.0.self_attn.k_proj", "vision_model.transformer.layers.2.self_attn.v_proj", "language_model.model.layers.14.mlp.up_proj", "vision_model.transformer.layers.24.self_attn.o_proj", "vision_model.transformer.layers.8.mlp.fc2", "language_model.model.layers.22.self_attn.o_proj", "vision_model.transformer.layers.3.self_attn.k_proj", "language_model.model.layers.14.self_attn.o_proj", "language_model.model.layers.23.cross_attn.k_proj", "language_model.model.layers.27.mlp.gate_proj", "language_model.model.layers.10.mlp.gate_proj", "vision_model.transformer.layers.21.self_attn.v_proj", "language_model.model.layers.29.self_attn.o_proj", "language_model.model.layers.31.self_attn.o_proj", "language_model.model.layers.38.cross_attn.o_proj", "vision_model.transformer.layers.9.self_attn.q_proj", "language_model.model.layers.32.mlp.gate_proj", "language_model.model.layers.6.self_attn.o_proj", "language_model.model.layers.24.mlp.up_proj", "language_model.model.layers.15.self_attn.k_proj", "vision_model.transformer.layers.20.self_attn.q_proj", "vision_model.transformer.layers.6.self_attn.k_proj", "vision_model.global_transformer.layers.4.self_attn.k_proj", "language_model.model.layers.22.mlp.down_proj", "language_model.model.layers.2.self_attn.o_proj", "language_model.model.layers.26.self_attn.o_proj", "language_model.model.layers.26.mlp.up_proj", "language_model.model.layers.37.self_attn.v_proj", "language_model.model.layers.14.self_attn.k_proj", "language_model.model.layers.25.self_attn.k_proj", "vision_model.transformer.layers.27.self_attn.k_proj", "vision_model.transformer.layers.25.self_attn.o_proj", "language_model.model.layers.9.mlp.down_proj", "language_model.model.layers.9.self_attn.k_proj", "vision_model.transformer.layers.9.self_attn.o_proj", "vision_model.transformer.layers.25.self_attn.v_proj", "vision_model.transformer.layers.26.mlp.fc2", "language_model.model.layers.29.self_attn.q_proj", "language_model.model.layers.9.self_attn.o_proj", "vision_model.transformer.layers.1.mlp.fc2", "language_model.model.layers.20.mlp.up_proj", "language_model.model.layers.11.self_attn.o_proj", "vision_model.transformer.layers.22.self_attn.k_proj", "language_model.model.layers.35.self_attn.v_proj", "language_model.model.layers.8.cross_attn.o_proj", "vision_model.transformer.layers.13.self_attn.k_proj", "vision_model.transformer.layers.11.mlp.fc2", "vision_model.transformer.layers.27.mlp.fc1", "vision_model.transformer.layers.29.self_attn.o_proj", "language_model.model.layers.33.cross_attn.q_proj", "vision_model.transformer.layers.0.mlp.fc2", "vision_model.transformer.layers.17.self_attn.k_proj", "language_model.model.layers.27.self_attn.q_proj", "vision_model.transformer.layers.14.mlp.fc1", "language_model.model.layers.4.mlp.gate_proj", "vision_model.transformer.layers.5.self_attn.v_proj", "language_model.model.layers.23.cross_attn.q_proj", "language_model.model.layers.6.mlp.gate_proj", "vision_model.transformer.layers.15.self_attn.k_proj", "language_model.model.layers.30.mlp.up_proj", "language_model.model.layers.29.self_attn.k_proj", "vision_model.global_transformer.layers.2.self_attn.v_proj", "language_model.model.layers.1.self_attn.o_proj", "vision_model.global_transformer.layers.5.self_attn.v_proj", "vision_model.global_transformer.layers.7.mlp.fc1", "vision_model.transformer.layers.22.mlp.fc1", "language_model.model.layers.0.mlp.up_proj", "vision_model.transformer.layers.27.mlp.fc2", "language_model.model.layers.24.self_attn.k_proj", "language_model.model.layers.25.mlp.up_proj", "language_model.model.layers.37.self_attn.q_proj", "language_model.model.layers.21.mlp.up_proj", "language_model.model.layers.22.mlp.gate_proj", "vision_model.transformer.layers.2.self_attn.k_proj", "vision_model.transformer.layers.9.self_attn.v_proj", "language_model.model.layers.10.mlp.up_proj", "language_model.model.layers.21.self_attn.v_proj", "vision_model.global_transformer.layers.3.mlp.fc1", "vision_model.transformer.layers.8.self_attn.v_proj", "vision_model.transformer.layers.12.self_attn.v_proj", "vision_model.transformer.layers.2.mlp.fc2", "language_model.model.layers.22.self_attn.k_proj", "vision_model.transformer.layers.5.mlp.fc2", "vision_model.transformer.layers.1.self_attn.q_proj", "language_model.model.layers.27.mlp.down_proj", "vision_model.transformer.layers.16.self_attn.k_proj", "language_model.model.layers.18.mlp.gate_proj", "vision_model.transformer.layers.24.mlp.fc2", "language_model.model.layers.38.mlp.gate_proj", "language_model.model.layers.27.self_attn.k_proj", "language_model.model.layers.33.mlp.down_proj", "language_model.model.layers.5.self_attn.o_proj", "language_model.model.layers.19.mlp.down_proj", "language_model.model.layers.1.self_attn.q_proj", "language_model.model.layers.25.mlp.down_proj", "vision_model.transformer.layers.8.self_attn.q_proj", "vision_model.transformer.layers.7.self_attn.v_proj", "language_model.model.layers.20.mlp.gate_proj", "vision_model.transformer.layers.7.self_attn.q_proj", "language_model.model.layers.30.mlp.down_proj", "vision_model.transformer.layers.30.self_attn.k_proj", "language_model.model.layers.9.mlp.gate_proj", "language_model.model.layers.8.mlp.up_proj", "vision_model.global_transformer.layers.1.self_attn.o_proj", "language_model.model.layers.38.mlp.up_proj", "vision_model.transformer.layers.25.mlp.fc1", "language_model.model.layers.12.mlp.down_proj", "vision_model.transformer.layers.8.mlp.fc1", "language_model.model.layers.7.self_attn.v_proj", "language_model.model.layers.4.mlp.up_proj", "language_model.model.layers.23.cross_attn.v_proj", "vision_model.transformer.layers.12.self_attn.k_proj", "language_model.model.layers.23.mlp.gate_proj", "vision_model.transformer.layers.20.mlp.fc2", "language_model.model.layers.2.mlp.gate_proj", "vision_model.gated_positional_embedding.tile_embedding", "language_model.model.layers.18.cross_attn.v_proj", "vision_model.global_transformer.layers.1.mlp.fc1", "vision_model.global_transformer.layers.4.self_attn.v_proj", "language_model.model.layers.37.mlp.gate_proj", "vision_model.transformer.layers.23.self_attn.o_proj", "language_model.model.layers.5.mlp.up_proj", "language_model.model.layers.31.self_attn.k_proj", "language_model.model.layers.31.self_attn.v_proj", "vision_model.transformer.layers.18.self_attn.k_proj", "language_model.model.layers.33.mlp.gate_proj", "vision_model.global_transformer.layers.5.self_attn.q_proj", "language_model.model.layers.2.mlp.up_proj", "vision_model.global_transformer.layers.3.self_attn.v_proj", "vision_model.global_transformer.layers.7.self_attn.o_proj", "language_model.model.layers.6.self_attn.q_proj", "language_model.model.layers.24.mlp.gate_proj", "vision_model.transformer.layers.24.mlp.fc1", "language_model.model.layers.9.self_attn.q_proj", "language_model.model.layers.10.self_attn.o_proj", "vision_model.transformer.layers.28.self_attn.v_proj", "vision_model.transformer.layers.29.self_attn.q_proj", "language_model.model.layers.39.self_attn.v_proj", "vision_model.transformer.layers.10.mlp.fc2", "vision_model.transformer.layers.25.mlp.fc2", "vision_model.transformer.layers.28.mlp.fc1", "vision_model.transformer.layers.21.self_attn.o_proj", "vision_model.transformer.layers.6.mlp.fc2", "vision_model.transformer.layers.19.self_attn.k_proj", "vision_model.transformer.layers.15.self_attn.q_proj", "vision_model.transformer.layers.18.self_attn.v_proj", "vision_model.global_transformer.layers.7.self_attn.q_proj", "vision_model.global_transformer.layers.1.self_attn.k_proj", "vision_model.global_transformer.layers.6.self_attn.v_proj", "language_model.model.layers.34.self_attn.q_proj", "language_model.model.layers.23.mlp.up_proj", "language_model.model.layers.29.mlp.down_proj", "vision_model.transformer.layers.23.self_attn.k_proj", "language_model.model.layers.21.self_attn.k_proj", "language_model.model.layers.30.self_attn.q_proj", "language_model.model.layers.31.mlp.gate_proj", "vision_model.transformer.layers.23.self_attn.v_proj", "language_model.model.layers.11.mlp.down_proj", "language_model.model.layers.15.self_attn.q_proj", "vision_model.transformer.layers.18.self_attn.q_proj", "vision_model.transformer.layers.27.self_attn.q_proj", "vision_model.transformer.layers.19.self_attn.o_proj", "language_model.model.layers.4.self_attn.q_proj", "language_model.model.layers.0.self_attn.q_proj", "vision_model.transformer.layers.13.mlp.fc2", "language_model.model.layers.18.mlp.up_proj", "language_model.model.layers.6.self_attn.k_proj", "language_model.model.layers.34.self_attn.k_proj", "vision_model.global_transformer.layers.3.self_attn.k_proj", "vision_model.global_transformer.layers.2.self_attn.q_proj", "vision_model.transformer.layers.11.self_attn.v_proj", "vision_model.transformer.layers.18.mlp.fc2", "language_model.model.layers.34.mlp.gate_proj", "vision_model.global_transformer.layers.3.mlp.fc2", "language_model.model.layers.17.mlp.gate_proj", "language_model.model.layers.35.mlp.down_proj", "vision_model.transformer.layers.4.mlp.fc2", "vision_model.global_transformer.layers.4.mlp.fc2", "language_model.model.layers.27.self_attn.v_proj", "language_model.model.layers.30.self_attn.o_proj", "language_model.model.layers.5.self_attn.k_proj", "language_model.model.layers.15.self_attn.o_proj", "vision_model.transformer.layers.30.self_attn.q_proj", "language_model.model.layers.17.self_attn.o_proj", "vision_model.transformer.layers.12.mlp.fc1", "vision_model.transformer.layers.16.self_attn.v_proj", "vision_model.transformer.layers.7.mlp.fc2", "language_model.model.layers.12.self_attn.v_proj", "vision_model.transformer.layers.17.self_attn.v_proj", "vision_model.transformer.layers.17.mlp.fc1", "vision_model.global_transformer.layers.5.mlp.fc1", "language_model.model.layers.35.self_attn.o_proj", "vision_model.global_transformer.layers.4.mlp.fc1", "language_model.model.layers.1.self_attn.k_proj", "language_model.model.layers.28.mlp.up_proj", "language_model.model.layers.3.mlp.up_proj", "language_model.model.layers.30.mlp.gate_proj", "vision_model.global_transformer.layers.0.mlp.fc2", "language_model.model.layers.25.self_attn.o_proj", "language_model.model.layers.10.self_attn.k_proj", "language_model.model.layers.0.mlp.down_proj", "language_model.model.layers.26.self_attn.k_proj", "vision_model.transformer.layers.21.self_attn.q_proj", "vision_model.global_transformer.layers.6.self_attn.q_proj", "vision_model.transformer.layers.19.self_attn.v_proj", "language_model.model.layers.11.mlp.up_proj", "language_model.model.layers.20.self_attn.v_proj", "language_model.model.layers.38.cross_attn.k_proj", "vision_model.transformer.layers.12.self_attn.q_proj", "language_model.model.layers.12.mlp.gate_proj", "vision_model.transformer.layers.12.mlp.fc2", "language_model.model.layers.36.mlp.gate_proj", "vision_model.pre_tile_positional_embedding.embedding", "vision_model.global_transformer.layers.3.self_attn.q_proj", "language_model.model.layers.32.mlp.down_proj", "vision_model.transformer.layers.6.mlp.fc1", "vision_model.transformer.layers.1.self_attn.o_proj", "vision_model.transformer.layers.23.mlp.fc2", "language_model.model.layers.20.self_attn.q_proj", "vision_model.transformer.layers.15.mlp.fc2", "language_model.model.layers.14.self_attn.v_proj", "vision_model.transformer.layers.18.mlp.fc1", "vision_model.global_transformer.layers.6.self_attn.o_proj", "language_model.model.layers.26.mlp.gate_proj", "language_model.model.layers.31.mlp.up_proj", "language_model.model.layers.14.mlp.gate_proj", "vision_model.transformer.layers.16.self_attn.q_proj", "language_model.model.layers.0.self_attn.v_proj", "language_model.model.layers.37.mlp.up_proj", "vision_model.transformer.layers.5.self_attn.o_proj", "language_model.lm_head", "language_model.model.layers.24.self_attn.q_proj", "language_model.model.layers.19.self_attn.q_proj", "language_model.model.layers.25.mlp.gate_proj", "vision_model.global_transformer.layers.3.self_attn.o_proj", "language_model.model.layers.39.mlp.up_proj", "language_model.model.layers.36.mlp.down_proj", "vision_model.transformer.layers.19.mlp.fc2", "language_model.model.layers.39.self_attn.k_proj", "language_model.model.layers.34.self_attn.v_proj", "vision_model.transformer.layers.11.self_attn.q_proj", "language_model.model.layers.22.self_attn.v_proj", "language_model.model.layers.29.mlp.gate_proj", "language_model.model.layers.39.self_attn.q_proj", "language_model.model.layers.14.mlp.down_proj", "vision_model.transformer.layers.2.self_attn.q_proj", "language_model.model.layers.6.mlp.down_proj", "vision_model.transformer.layers.4.self_attn.v_proj", "vision_model.transformer.layers.17.self_attn.o_proj", "vision_model.transformer.layers.13.self_attn.q_proj", "language_model.model.layers.29.mlp.up_proj", "vision_model.global_transformer.layers.0.self_attn.v_proj", "language_model.model.layers.17.mlp.up_proj", "language_model.model.layers.24.self_attn.o_proj", "vision_model.transformer.layers.1.self_attn.v_proj", "language_model.model.layers.30.self_attn.k_proj", "language_model.model.layers.17.self_attn.q_proj", "vision_model.global_transformer.layers.2.mlp.fc2", "vision_model.global_transformer.layers.0.mlp.fc1", "vision_model.transformer.layers.4.self_attn.k_proj", "language_model.model.layers.7.mlp.up_proj", "vision_model.transformer.layers.6.self_attn.q_proj", "language_model.model.layers.36.self_attn.q_proj", "language_model.model.layers.10.mlp.down_proj", "language_model.model.layers.17.mlp.down_proj", "vision_model.transformer.layers.23.self_attn.q_proj", "language_model.model.layers.35.mlp.up_proj", "vision_model.transformer.layers.21.mlp.fc1", "vision_model.transformer.layers.31.self_attn.k_proj", "language_model.model.layers.15.mlp.gate_proj", "language_model.model.layers.8.cross_attn.v_proj", "language_model.model.layers.13.cross_attn.k_proj", "vision_model.transformer.layers.19.self_attn.q_proj", "language_model.model.layers.2.self_attn.v_proj", "vision_model.global_transformer.layers.0.self_attn.q_proj", "language_model.model.layers.27.self_attn.o_proj", "vision_model.transformer.layers.15.mlp.fc1", "vision_model.transformer.layers.20.self_attn.v_proj", "language_model.model.layers.32.self_attn.v_proj", "vision_model.transformer.layers.6.self_attn.v_proj", "language_model.model.layers.32.self_attn.q_proj", "vision_model.transformer.layers.22.mlp.fc2", "language_model.model.layers.28.cross_attn.v_proj", "vision_model.transformer.layers.29.mlp.fc2", "vision_model.global_transformer.layers.1.self_attn.q_proj", "vision_model.global_transformer.layers.6.mlp.fc1", "language_model.model.layers.2.self_attn.k_proj", "language_model.model.layers.8.cross_attn.k_proj", "language_model.model.layers.31.self_attn.q_proj", "vision_model.transformer.layers.7.self_attn.k_proj", "vision_model.transformer.layers.28.self_attn.k_proj", "vision_model.transformer.layers.26.self_attn.o_proj", "vision_model.transformer.layers.10.self_attn.k_proj", "language_model.model.layers.32.self_attn.k_proj", "language_model.model.layers.6.mlp.up_proj", "vision_model.transformer.layers.28.self_attn.q_proj", "language_model.model.layers.18.cross_attn.o_proj", "language_model.model.layers.13.mlp.down_proj", "language_model.model.layers.12.self_attn.q_proj", "language_model.model.layers.19.self_attn.k_proj", "language_model.model.layers.20.mlp.down_proj", "language_model.model.layers.4.self_attn.o_proj", "language_model.model.layers.5.mlp.gate_proj", "language_model.model.layers.18.cross_attn.k_proj", "vision_model.transformer.layers.15.self_attn.o_proj", "language_model.model.layers.16.mlp.up_proj", "vision_model.transformer.layers.8.self_attn.k_proj", "vision_model.transformer.layers.3.self_attn.q_proj", "language_model.model.layers.36.self_attn.v_proj", "vision_model.transformer.layers.9.mlp.fc2", "vision_model.transformer.layers.17.mlp.fc2", "language_model.model.layers.4.self_attn.v_proj", "vision_model.transformer.layers.10.self_attn.o_proj", "vision_model.transformer.layers.27.self_attn.v_proj", "vision_model.global_transformer.layers.2.self_attn.k_proj", "language_model.model.layers.8.mlp.down_proj", "language_model.model.layers.21.mlp.gate_proj", "language_model.model.layers.33.cross_attn.k_proj", "vision_model.transformer.layers.10.self_attn.v_proj", "language_model.model.layers.24.self_attn.v_proj", "language_model.model.layers.38.mlp.down_proj", "vision_model.transformer.layers.11.mlp.fc1", "vision_model.global_transformer.layers.0.self_attn.k_proj", "language_model.model.layers.32.mlp.up_proj", "vision_model.transformer.layers.3.self_attn.o_proj" ], "task_type": null, "use_dora": false, "use_rslora": false } adapter_config.json

[NandhaKishorM]

Traceback (most recent call last): File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/foura100gpu/code/Users/nandakishor/7bllava/adataset/Llama3.2-Vision-Finetune/src/merge_lora_weights.py", line 21, in merge_lora(args) File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/foura100gpu/code/Users/nandakishor/7bllava/adataset/Llama3.2-Vision-Finetune/src/merge_lora_weights.py", line 5, in merge_lora processor, model = load_pretrained_model(model_path=args.model_path, model_base=args.model_base, File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/foura100gpu/code/Users/nandakishor/7bllava/adataset/Llama3.2-Vision-Finetune/src/utils.py", line 48, in load_pretrained_model token_num, tokem_dim = model.language_model.lm_head.out_features, model.language_model.lm_head.in_features File "/anaconda/envs/azureml_py38/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1695, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'MllamaForCausalLM' object has no attribute 'language_model'

[2U1]

Sorry, my bad. It was the problem for loading the model. I had to use MllamaForConditionalGeneration. I've fixed the issue. You could try it again.

Your issue helped me a lot. I really appriciate it. Thanks

[NandhaKishorM]

thanks, it worked