search

derrian-distro / LoRA_Easy_Training_Scripts

A UI made in Pyside6 to make training LoRA/LoCon and other LoRA type models in sd-scripts easy
GNU General Public License v3.0
998 stars 101 forks source link

BUG? REX LR-Scheduler not being set correclty? (current dev-branch) #193

Closed neojam closed 3 months ago

neojam commented 3 months ago

Hi there,

i just wanted to try out the CAME Otimizer + REX scheduler, but it looks like the REX scheduler is not being set correctly (according to toml and meta of the lora. Im on the dev branch btw.)

"lr_scheduler" in toml file is always being set to whatever scheduler you choose in the scheduler dropdown menu, before selecting it to "rex". So if you selected "polynomial" and then selected the "rex", the settings are saved as following in the toml-file

lr_scheduler = "polynomial"
lr_scheduler_type = "LoraEasyCustomOptimizer.CustomOptimizers.Rex"

If you bake the lora, the meta will say

"ss_lr_scheduler":"polynomial"

I got interested in came+rex (and so in this project, since apparently other "kohya forks" do not have that) after checking the meta of few good trained loras on civitai. They have the following in the meta:

"ss_lr_scheduler":"REX",
"ss_optimizer":"came_pytorch.CAME.CAME(weight_decay=0.04)",

But i cant match the settings, since i cant select the scheduler to "REX".

Any Idea whats going on?

Also maybe as a side request: It would be great, if there were was a way to set the settings directly from lora's metadata (Lora-meta to Toml tool, or just directly opening a lora in LoRA_Easy_Training_Scripts GUI and have the most settings already set).

Jelosus2 commented 3 months ago

Hi there, 2 points.

  1. The lr scheduler type name is assigned to other scheduler (like polynomial, cosine, etc) because the REX scheduler is not added into the sd scripts which are the scripts that makes the training possible. In this trainer you can create your custom schedulers and make them available for sd scripts to use without needing to modify the scripts code itself. As long as the lr_scheduler_type is REX it should be fine.
  2. Let me explain why some people using this trainer on their LoRA metadata they have REX, I would also like if you could tell me the names of the creator(s) to verify what I am saying. They used a modified version of this trainer that I and other people made that modified the code of sd scripts to add REX scheduler since we wanted to use CAME and REX before Derriana published his official addition of them. Also if you notice in the ss_optimizer you posted it starts with came_pytorch which is the library we used in our modification. The library that the implementation of Derrian's uses is pytorch_optimizer

I hope this helps you to understand why its different. If you have any other question don't hesitate to ask :)

neojam commented 3 months ago

Hi thanks for the info. That sounds both interesting and confusing ^^'. So is there another fork of this trainer?

Here is the concept Lora that works pretty well, so i wanted to try out the settings for my bakes: https://civitai.com/models/332691/broken-glass-breaking-glass-or-concept-lora-xl

Jelosus2 commented 3 months ago

Hi thanks for the info. That sounds both interesting and confusing ^^'. So is there another fork of this trainer?

Here is the concept Lora that works pretty well, so i wanted to try out the settings for my makes: https://civitai.com/models/332691/broken-glass-breaking-glass-or-concept-lora-xl

Yup, FallenIncursio used (or still uses) the modified version of the trainer. But don't worry, the dev branch uses REX properly and the CAME from pytorch_optimizer should be better than the one from came_pytorch

neojam commented 3 months ago

ok, thanks for the explanation.

IMHO the "lr_scheduler" in dev branch still needs a change to reflect the fact that REX was used in "lr_scheduler_type". As it currently is (setting it to previously selected scheduler for toml and meta) seems wrong and will give false info about what scheduler was really used during the training for people who check meta and want to try the settings.

Jelosus2 commented 3 months ago

ok, thanks for the explanation.

IMHO the "lr_scheduler" in dev branch still needs a change to reflect the fact that REX was used in "lr_scheduler_type". As it currently is (setting it to previously selected scheduler for toml and meta) seems wrong and will give false info about what scheduler was really used during the training for people who check meta and want to try the settings.

Derrian doesn't want to modify the sd scripts code, unless there's a way to add that without touching its code it won't be implemented

neojam commented 3 months ago

AFAIK, the meta can be easily changed post training, so maybe use it as a workaround to change "ss_lr_scheduler" to "REX" (and maybe later for other things as well, if more non-standard features get implemented). But for now I guess the Issue can be closed.

derrian-distro commented 3 months ago

I don't think it's a worthwhile use of my time to reinvent the wheel, so I'll be closing this issue now