Recommendation: when training on Augmentoolkit data, use GaLore, NOT LoRAs

e-p-armstrong / augmentoolkit

Convert Compute And Books Into Instruct-Tuning Datasets

MIT License

584 stars 79 forks source link

Closed bodybreaker closed 2 weeks ago

bodybreaker commented 1 month ago

Hi, Is there any reason for "Recommendation: when training on Augmentoolkit data, use GaLore, NOT LoRAs"

e-p-armstrong commented 1 month ago

GaLore is like a full finetune; it's far more effective for teaching models factual information. LoRAs are mostly for stylistic adjustments.