Question about LoRA Target in Fine-tuning of Idefics2

[kash203]

It appears that the image encoder is not included as a target for LoRA in the current fine-tuning of Idefics2. Is there any specific reason for this, such as the low effectiveness of LoRA on the image encoder?

I would appreciate it if you could provide any insights on this matter.

Best regards,

Referenced file

https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Idefics2/Fine_tune_Idefics2_for_JSON_extraction_use_cases_(PyTorch_Lightning).ipynb

Relevant section

    if USE_ADD_ADAPTER:
        lora_config = LoraConfig(
            r=8,
            lora_alpha=8,
            lora_dropout=0.1,
            target_modules=".*(text_model|modality_projection|perceiver_resampler).*(down_proj|gate_proj|up_proj|k_proj|q_proj|v_proj|o_proj).*$",
            use_dora=False if USE_QLORA else True,
            init_lora_weights="gaussian",
        )
        model.add_adapter(lora_config)
        model.enable_adapters()

[NielsRogge]

Pinging @VictorSanh here, I took this from the DocVQA notebook