search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.67k stars 136 forks source link

Added automation tab with folder and mustache support #307

Closed gnewtzie closed 4 months ago

gnewtzie commented 4 months ago

Found myself constantly going in and changing the same values over and over as i create a lot of lora's with the same settings.

The new tab support pointing at a folder and for each folder it will run to completion and then start the next one. Each folder get's it's only copy of the config, concepts and samples files with the values replaced if necessary.

You can name the folder with dashes to support multiple mustache replacements throughout the settings.

Specifically it supports the workspace / cache / modeloutput / concepts / samples folder replacements.

An example would be /directory/{{V1}}{{V2}.safetensors in the model output and then name the folder "foo:bar" to get /directory/foobar.safetensors in the final config which is used as the "function_train_config". The original becomes a template. you can turn it off an on in the automation tab.

Nerogar commented 4 months ago

I like the general idea of this change. But to be honest, the actual implementations isn't great.

There are quite a few problems:

  1. The code quality is pretty low. You added several unused imports. Variables are assigned and never used. Function return types are not specified correctly, and so on.
  2. You added a new dependency just for string replacement. It should be possible to do the same with built in python functions.
  3. Usage is pretty complicated if you don't already know what you are doing. The way you implemented it, the user needs to built their own complicated directory structure, instead of using the UI for everything. I generally don't like having "magic" folder name conventions that aren't even documented anywhere.
  4. Queueing up work was already possible by exporting multiple configs, then calling scripts/train.py from a batch/shell script. You could have done something similar here. Add a new "queue" button that exports the current settings into some kind of work queue, and a new "run queue" button that sequentially executes the exported configs.
gnewtzie commented 4 months ago

spent more than a few hours on this today and i see your point. there was no need to include it into the main branch at all. scripts does everything i want without having to conform to the existing project structure and limitations to the existing build.

  1. it's my first python code i've ever written. threw it together in a few hours just to automate my workflow. so i understand. copilot was my friend during most of it.
  2. moustache is pretty much the standard way of doing string variable replacement in every other language so i figure it would work here. it's already documented and quite frankly i don't have to write it to deal with all the crap involved in string replacement. (i use it in prob 5 other languages).
  3. the folder structure was by design. i'm trying to avoid mouse clicks not add more. by being able to just look at the folder name i can immediately tell what params i passed into the replacement because i was changing a lot during the configuration. saves me having to open a variable file to see what i was thinking for that particular iteration. i may run the same lora and dataset 5 or 6 times to get what i want and by looking directly at the folder i can tell without 5 or 6 more clicks. Just my interpretation of automation. I'm all about speed.
  4. I kind of knew about this but never really dug deep into into it. Once i did, i agree it's not really a good fit for the main branch. It's really a stand alone program that calls your scripts. i have most of it working with templates, folder name replacement (i'm adding a variable file if you want), auto captioning (may need a small tweak in the script here) and folder watching so you don't have to shut it down, just leave it running and drop a folder into the queue dir and it will pick it up when the folder size stablizes.
    Anyway, this has been fun doing something in python. worked with all these SD projects but never done anything for real in it. It reminds me of typescript for the most part but a slight bit more structured. (i'm mostly c# and business apps). I'll close it out with comment. sorry to bother you, thought it was useful to the codebase but i'll release it separately. Prob automate it against kohya too just for coverage.