Closed awsaf49 closed 7 months ago
Hi @awsaf49, since GaLore
is relatively new, we will likely wait on integrating this into the core API, but you are welcome to prototype this as a subclass of EinsumDense or Dense layer and keep us updated! Thanks.
Recently
GaLore
has been released which can be used for memory efficient fine-tuning of llms. According to the paper,I think it would be nice to have this feature in Keras 3.
Reference