search

GraphPKU / PiSSA

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models(NeurIPS 2024 Spotlight)
https://arxiv.org/abs/2404.02948
261 stars 9 forks source link

Why does my PiSSA init process so slow? #19

Open ANYMS-A opened 4 months ago

ANYMS-A commented 4 months ago

Hey, I recently tried to use PiSSA init during the LoRA finetuing process, however I noticed that the time takes to init with PiSSA is so long. I set all linear layer as my LoRA target layer and the Rank of the LoRA is 32. I used "pissa_niter_4" as the init_lora_weights argument, but seems it still takes about 20-30 minutes, is there any solutions for this ? Many thanks!

fxmeng commented 4 months ago

The execution time of (Fast) SVD is related to the dimensions of the matrices. Could you please specify which model you are using?

ANYMS-A commented 3 months ago

Hey thanks for the reply. I used the DeepSeek-v1.5 for the finetuning. One thing I observed is that: if I set the 'bias' as 'lora-only' when using pissa, the unit process will be slow, but if I set it to none, the init process of pissa will be much faster. BTW I use a Lora rank of "32"