search

GraphPKU / PiSSA

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models(NeurIPS 2024 Spotlight)
https://arxiv.org/abs/2404.02948
265 stars 10 forks source link

fast svd #25

Open hhnqqq opened 3 weeks ago

hhnqqq commented 3 weeks ago

我在peft里面似乎没有找到快速svd的代码 这是我自己用的代码，不知道是否和论文里的一致

            V, S, Uh = randomized_svd(self.weight.data.numpy(), n_components=self.lora_rank, random_state=None)
            Vr, Sr, Ur = map(torch.from_numpy, (V, S, Uh))
hhnqqq commented 3 weeks ago

另外，论文里的 Table 4: Comparation between SVD and Fast SVD in terms of initialization time, error and training loss有点看不懂

hhnqqq commented 3 weeks ago

我懂了，设置n_iters的话就是fast svd