search

kohya-ss / sd-scripts

Apache License 2.0
5.19k stars 862 forks source link

GLoRA: Generalized LoRA for Parameter-Efficient Fine-tuning #600

Open slashedstar opened 1 year ago

slashedstar commented 1 year ago

https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA

Since I don't have the technical knowledge to know how meaningful or relevant this is for lora in SD I'm posting it here to bring attention to it

DarkAlchy commented 1 year ago

That is a big one to get as it more automates the process. From what I am reading, I would love to see it in action on SD 2.1.

dmarx commented 1 year ago

Bumping this, didn't realize there was already a thing. I accidentally made a duplicate in #654 . I think it would be interesting and useful to implement GLoRA at least as a parameterization of the procedure, even without the evolutionary search piece. It's a generalization over LoRA and several other procedures. People might enjoy the flexibility of several additional new methods to play with (VPT, SSF, FacT, AdaptFormer...), which GLoRA gives you for free.

oliverban commented 1 year ago

Is this ever going to be a thing? reading the paper it seems to be a nice addition!

coding-famer commented 11 months ago

I'd like to work on this if no one else is doing this.

DarkAlchy commented 11 months ago

I wish I had the tech ability myself but I have to say with the way Kohya works it makes a mess of SDXL due to not handling both TEs as separate entities. Onetrainer has that in their beta and it is a game changer, and add this Kohya will be left in the dust.