search

jiaweizzhao / GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Apache License 2.0
1.24k stars 131 forks source link

Please add Phi-2 Support #19

Open calebmor460 opened 3 months ago

calebmor460 commented 3 months ago

Attempting to use Galore to finetune a phi model yields "AttributeError: 'PhiConfig' object has no attribute 'rms_norm_eps'", which, having gotten that error on other LLM things, typically translates to "this code doesn't support Phi models"

fixing this would be incredibly nice, as It would allow people with cruddier computers to finetune LLMs

jiaweizzhao commented 3 months ago

Can you share a bit more details about the problem you are facing? If you want to try fine-tuning Phi-2 using GaLore, I would suggest you to use https://github.com/hiyouga/LLaMA-Factory, which already integrated GaLore inside.