Meeting Notes 04/05

[sadmankiba]

• 2 people work on understanding and reimplementing LTE paper (Low-rank on weights for pre-training)
• 2 people work on understanding and reimplementing GaLoRE paper (Low-rank on gradients for pre-training)
• Report performance of both techniques in our project
• If we have time, we can combine the ideas from both paper, to come up with novel design and report in the project
• Visualize LoRA updates according to "Low-rank simplicity bias in deep networks"

[sadmankiba]

LTE - https://github.com/minyoungg/LTE GaLore - https://github.com/jiaweizzhao/GaLore

[sadmankiba]

A starting point NanoGpt-LoRA - https://github.com/danielgrittner/nanoGPT-LoRA