Question about orthogonal regularization methods

QingruZhang / AdaLoRA

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).

MIT License

259 stars 28 forks source link

Question about orthogonal regularization methods #22

Open fei407 opened 7 months ago

fei407 commented 7 months ago

Hi, thanks for this awesome work! I have a question regarding the two different orthogonal regularization methods, frd and svd, as they appear to have distinct impacts. How is this aspect considered in AdaLoRa?

QingruZhang commented 6 months ago

Hello, frd is implemented for LoRA, which regulaize A and B orthogonal but does not normalize. We present an ablation to compare AdaLoRA and LoRA with orthogonal regularization in Section 4.4 to show both components of AdaLoRA are important. For AdaLoRA, svd should be selected which regulalize P and Q both orthogonal and normalized.