Remove unused `A` and `B` computation

jiaweizzhao / GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Apache License 2.0

1.24k stars 131 forks source link

Remove unused `A` and `B` computation #18

Closed awgu closed 1 month ago

awgu commented 3 months ago

For type == "right", we do not need to compute A, and for type == "left", we do not need to compute B. (The variables are unused.) We can avoid this computation to save some FLOPs.

jiaweizzhao commented 1 month ago

Sorry for the late merge and thanks for the contribution!