search

bmaltais / kohya_ss

Apache License 2.0
9.71k stars 1.25k forks source link

improvment to the lora.py using ring architecture #2928

Open luckiskind opened 1 month ago

luckiskind commented 1 month ago

lora.txt lora_flux.txt used this paper to implement the basic methodology into the lora.py network https://github.com/DAMO-NLP-SG/Inf-CLIP

network dim 32 sdxl now maintains a speed of 3.4 sec/it at a batch size of 20 for less than 24gb on a 4090. my flux implementation needs some help. i managed to get a batch size of 3 with no split on dim 32. using adafactor for both. please take a look let me know if i can help in any way.

bmaltais commented 1 month ago

You should bring this up to kohya_ss on his sd-scripts repo. I only leverage his work... he is the one that need to make the improvements ;-)

luckiskind commented 1 month ago

thanks, where would i find the slider max value for batch size? im at 64 and still going.