8-bit Adam vs 32-bit Adam ??

vihangd / alpaca-qlora

Instruct-tune Open LLaMA / RedPajama / StableLM models on consumer hardware using QLoRA

Apache License 2.0

80 stars 11 forks source link

Open apachemycat opened 1 year ago

apachemycat commented 1 year ago

qlory triaining using paged_adamw_32bit but you using paged_adamw_8bit here maybe a problem ?