search

GraphPKU / PiSSA

PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models(NeurIPS 2024 Spotlight)
https://arxiv.org/abs/2404.02948
261 stars 9 forks source link

Training scripts of Pissa with LLaMA-2-7b for 3 epoches #18

Closed lucasliunju closed 4 months ago

lucasliunju commented 4 months ago

Hi,

I found the results on Figure 4(c) is about 3 epoches. May I ask the training scripts about this experiment or some important hyper-parameters, such as learning rate, rank, batch size?

fxmeng commented 4 months ago

The hyperparameters for the 3-epoch experiment are exactly the same as those provided in the project script. You just need to use all the training data and change the number of epochs to 3.