[Feature Request] GLoRA - "One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning"

[dmarx]

I think it would be interesting and useful to implement GLoRA at least as a parameterization of the procedure, even without the evolutionary search piece. It's a generalization over LoRA and several other procedures. People might enjoy the additional flexibility of several additional new methods to play with (VPT, SSF, FacT, AdaptFormer, RepAdaptor...), which GLoRA gives you for free.

• https://sites.google.com/view/generalized-lora
• https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA
• https://arxiv.org/abs/2306.07967

[KohakuBlueleaf]

already working on actually.

[KohakuBlueleaf]

the problem is inference

[KohakuBlueleaf]

BUT I think I will implement GLoKr at first, which have more fun to me :P

[KohakuBlueleaf]

@dmarx actually I already done the glora for a while I just forgot it............