how to define target modules in the Qlora script

DAMO-NLP-SG / VideoLLaMA2

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

Apache License 2.0

871 stars 60 forks source link

how to define target modules in the Qlora script #67

Closed thisurawz1 closed 2 months ago

thisurawz1 commented 3 months ago

How do you use all linear layers with LoRA adapters? like q_proj, k_proj, v_proj, o_proj, w1, w2, w3, and lm_head. and what are the layers that are finetuned with the current QLoRA script?

clownrat6 commented 2 months ago

Except vision encoder, all of the layers will be finetuned. The training parameters of Lora or QLora are same.

thisurawz1 commented 2 months ago

ok understood. thanks