search

Clin0212 / HydraLoRA

[NeurIPS'24 Oral] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
72 stars 5 forks source link

lora_target #7

Closed lightest1 closed 3 days ago

lightest1 commented 4 days ago

Hello! Thank you for your work! From the code, it seems that lora only acts on gate_proj,down_proj and up_proj. I want to know whether the result in the paper is the result of these lora_targets, or whether the code should be modified into q_proj, k_proj or v_proj?

Clin0212 commented 4 days ago

Thank you for your question! Yes, in this case, LoRA is applied only to the linear layers in the FFN (gate_proj, down_proj, up_proj). However, you can modify the code to include q_proj, k_proj, or v_proj (which are related to the self-attention layer) as lora_targets based on your specific needs.