Huge Changes Incoming

[Linaqruf]

I am working on updating Kohya's script to the latest version. The current version is from February 3rd, but the latest version in sd-scripts is from February 11th. This update will bring many new arguments and changes. I am also restructuring the notebooks to make the code more readable, maintainable, and user-friendly.

Here are some of the changes I have made:

1. I have switched from using git-large-textcaps for auto captioning to using blip captioning with beam search enabled. The reason being that git is slower, while blip with beam search has similar quality.
2. . I have deleted the Image Upscaler with R-ESRGAN cell because the bucketing already upscales images when converting to latents.
3. I have combined the Login to Huggingface Hub cell with the cell where you define the model and dataset repository, making the workflow easier.
4. New Arguments (useful for advanced users, but may confuse end-users):

• bucket_reso_steps = 64 #@param {type:"slider", min:0, max:100, step:8}
• bucket_no_upscale = False #@param{type:"boolean"}
• caption_dropout_rate = 0 #@param {type:"number"}
• caption_dropout_every_n_epochs = 0 #@param {type:"number"}
• tag_dropout_rate = 0 #@param {type:"number"}

5.I have chosen not to use these and let advanced users register new arguments in additional_arguments.

6. have combined the data cleaning cell with the RGB -> RGBA converter.
7. I have improved the model converter cell (in both the main and dreambooth notebooks).

Here's a sneak peek: https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/8c52bc700f8cbb68a6ee58e4f73e27067d885e5d/kohya-LoRA-dreambooth.ipynb

[cdefghijkl]

It would be helpful if you can write on there a little about the function of those arguments.

[noriwaru]

I get an error when I try to use 「bucket_reso_steps」and 「bucket_no_upscale」

train_network.py: error: unrecognized arguments: --bucket_no_upscale Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/accelerate_cli.py", line 45, in main args.func(args) File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 1104, in launch_command simple_launcher(args) File "/usr/local/lib/python3.8/dist-packages/accelerate/commands/launch.py", line 567, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

[Linaqruf]

you need to input dev to branch field

[Linaqruf]

It would be helpful if you can write on there a little about the function of those arguments.

you can read it here, https://github.com/kohya-ss/sd-scripts

[shinkarom]

Can you add unet scale and text encoder scale? According to https://github.com/cloneofsimo/lora/discussions/37 , in original repo both items can be changed separately, and not having them at 1.0 can sometimes be better.

[noriwaru]

Thanks for your hard work.

[Linaqruf]

v13 (25/02):

What Changes?

• Of course refactoring, cleaning and make the code and cells more readable and easy to maintain.
  • Moved Login to Huggingface Hub to Deployment section, in the same cell with defining repo.
  • Merged Install Kohya Trainer, Install Dependencies, and Mount Drive cells
  • Merged Dataset Cleaning and Convert RGB to RGBA cells
  • Deleted Image Upscaler cell, because bucketing automatically upscale your dataset (converted to image latents) to min_bucket_reso value.
  • Deleted Colab Ram Patch because now you can set --lowram in the training script.
  • Revamped Unzip dataset cell to make it look simpler
• Added xformers pre-compiled wheel for A100
• Revamped Pretrained Model section
  • Deleted some old pretrained model
  • Added Anything V3.3, Chilloutmix, and Counterfeit V2.5 as new pretrained model for SD V1.x based model
  • Added Replicant V1.0, WD 1.5 Beta and Illuminati Diffusion V1 as new pretrained model for SD V2.x 768v based model
  • Changed Stable Diffusion 1.5 pretrained model to pruned one.
• Changed Natural Language Captioning back from GIT to BLIP with beam_search enabled by default
• Revamped Image Scraper from simple to advanced, added new feature such as:
  • Added safebooru to booru list
  • Added custom_url option, so you can copy and paste the url instead of specify which booru sites and tags to scrape
  • Added user_agent field, because you can't access some image board with default user_agent
  • Added limit_rate field to limit your count
  • [Experimental] Added with_aria2c to scrape your dataset, not a wrapper, just a simple trick to extract urls with gallery-dl and download them with aria2c instead. Fast but seems igonoring --write-tags.
  • All downloaded tags now saved with .txt format instead of .jpg.txt
  • Added additional_arguments to make it more flexible if you want to try other args
• Revamped Append Custom Tag cell
  • Create new caption file for every image file based on extension provided (.txt/.caption) if you didn't want to use BLIP or WD Tagger
  • Added --keep_tokens args to the cell
• Revamped Training Model section.
  • Revamped prettytable for easier maintenance and bug fixing
  • Now it has 4 major cell:
  • Folder Config
    • To specify v2, v2_parameterization and all important folder and project_name
  • LoRA and Optimizer Config
    • Only Optimizer Config for notebook outside LoRA training
    • All about Optimizer, learning_rate and lr_scheduler goes here
    • Added new Optimizer from latest kohya-ss/sd-script, all available optimizer : `"AdamW", "AdamW8bit", "Lion", "SGDNesterov", "SGDNesterov8bit", "DAdaptation", "AdaFactor"
    • Currently you can't use DAdaptation if you're in Colab free tier because it need more VRAM
    • Added --optimizer_args for custom args, useful if you want to try adjusting weight decay, betas etc
  • Dataset Config
    • Only available for Dreambooth method notebook, it basically bucketing cell for Dreambooth.
    • Added caption dropout, you can drop your caption or tags by adjusting dropout rates.
    • Added --bucket_reso_steps and --bucket_no_upscale
  • Training Config
    • Added --noise_offset, read Diffusion With Offset Noise
    • Added --lowram to load the model in VRAM instead of CPU
• Revamped Convert Diffusers to Checkpoint cell, now it's more readable.
• Fixing bugs when output_dir located in google drive, it assert an error because of something like /content/drive/dreambooth_cmd.yaml which is forbidden, now instead of saved to {output_dir}, now training args history are saved to {training_dir}

News

• I'm in burnout phase, so I'm sorry for the lame update.
• Fast Kohya Trainer, an idea to merge all Kohya's training script into one cell. Please check it here.
  • Please don't expect high, it just a secondary project and maintaining 1-click cell is hard. So I won't prioritized it.
• Kohya Textual Inversion are cancelled for now, because maintaining 4 Colab Notebook already making me this tired.
  • Please use this instead, not kohya script but everyone on WD server using this since last year:
  • stable-textual-inversion-cafe colab
  • stable-textual-inversion-cafe Colab - Lazy Edition
• I wrote a Colab Notebook for #AUTOMATIC1111's #stablediffusion Web UI, with built-in Mikubill's #ControlNet extension. All Annotator and extracted ControlNet model are provided in the notebook. It's called Cagliostro Colab UI. Please try it.
  • You can use new UI/UX from Anapnoe in the notebook. You can find the option in experimental section.

[Linaqruf]

Please let me know if there is bugs, and error.

[younyokel]

pretty confusing fr

[Linaqruf]

Ikr, the trainer is more advanced than before, you can use old commit notebook or just train with default value if you find it too hard.

All new optimizer types, args, offset noise, caption dropout, new bucketing option, etc, too much new args to try, but not much time to try all of them.

[Linaqruf]

I didn't change anything about default value nor backend script (except tag reading) compared to the previous version. You need to provide a screenshot, training logs, or copied notebook with the same hyperparams and dataset so I can figure out what is the error.

And yes, max_train_epochs is not that accurate.

You can complain about backend script here instead : https://github.com/kohya-ss/sd-scripts

[Linaqruf]

something not passed to the accelerate, 1680 was the default maxsteps

i think this is because the linebreak \ in the training config

can you check the train.sh to see all hyperparams passed?

[Linaqruf]

i need to change this to yaml asap