Training with QLORA - Githubissues

microsoft / DeepSpeedExamples

Example models using DeepSpeed

Apache License 2.0

5.83k stars 990 forks source link

Open puyuanOT opened 1 year ago

puyuanOT commented 1 year ago

Is there a way to utilize DeepSpeed for QLORA training? It looks like QLORA requires a special optimizer (e.g., paged_adamw_8bit).

Nisoka commented 12 months ago

want too!

elaaaf commented 10 months ago

+++