search

QingruZhang / AdaLoRA

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
MIT License
231 stars 23 forks source link

question about early stopping #3

Closed DopamineLcy closed 1 year ago

DopamineLcy commented 1 year ago

Thank you for your marvelous work! Did you use a validation set and early stopping to select the best-performing checkpoint? If so, how do you ensure that the budget reaches the target budget? Best,

QingruZhang commented 1 year ago

Hi, thanks for your question. We did not apply early stopping. We evaluate the model during the final fine-tuning stages, when AdaLoRA reaches the target budget and fix it. The checkpoint can be selected during the final fine-tuning stage, i.e. after total_step-final_warmup steps. Hope this can answer your questions.

A11en0 commented 11 months ago

Why not use an early stop, otherwise it may not find the best checkpoint. And Adalora seems not select the best checkpoint in the final fine-tuning stage.