LoRa Training - Can't get black and white images

[synapsestorm]

I'm training a LoRa that has a kind of black and white/sepia and white style. I've spent many many hours training and messing around with different settings, but can't ever get pure black and white/sepia and white results, they always have some sort of random colour to them. The data set is purely b&w images with no colour to them other than the slight brownness, like the page of a book.

Settings: { "pretrained_model_name_or_path": "D:/stable-diffusion-webui/models/Stable-diffusion/nai.ckpt", "v2": false, "v_parameterization": false, "logging_dir": "D:/Kohya/kohya_ss/Lora training data/MyFolder/log", "train_data_dir": "D:/Kohya/kohya_ss/Lora training data/MyFolder/image", "reg_data_dir": "", "output_dir": "D:/Kohya/kohya_ss/Lora training data/MyFolder/model", "max_resolution": "576,576", "learning_rate": "0.0005", "lr_scheduler": "cosine_with_restarts", "lr_warmup": "0", "train_batch_size": 10, "epoch": 15, "save_every_n_epochs": 1, "mixed_precision": "bf16", "save_precision": "bf16", "seed": "678452056", "num_cpu_threads_per_process": 2, "cache_latents": true, "caption_extension": ".txt", "enable_bucket": true, "gradient_checkpointing": false, "full_fp16": false, "no_token_padding": false, "stop_text_encoder_training": 0, "xformers": true, "save_model_as": "safetensors", "shuffle_caption": false, "save_state": false, "resume": "", "prior_loss_weight": 1.0, "text_encoder_lr": "0.0003", "unet_lr": "0.0005", "network_dim": 128, "lora_network_weights": "", "color_aug": false, "flip_aug": true, "clip_skip": 2, "gradient_accumulation_steps": 1.0, "mem_eff_attn": false, "output_name": "MyLoraName", "model_list": "custom", "max_token_length": "75", "max_train_epochs": "", "max_data_loader_n_workers": "1", "network_alpha": 64, "training_comment": "", "keep_tokens": 1, "lr_scheduler_num_cycles": "", "lr_scheduler_power": "", "persistent_data_loader_workers": false, "bucket_no_upscale": true, "random_crop": false, "bucket_reso_steps": 64.0, "caption_dropout_every_n_epochs": 0.0, "caption_dropout_rate": 0, "optimizer": "AdamW8bit", "optimizer_args": "", "noise_offset": "0.06", "LoRA_type": "Standard", "conv_dim": 1, "conv_alpha": 1, "sample_every_n_steps": 50, "sample_every_n_epochs": 0, "sample_sampler": "k_dpm_2", "sample_prompts": "MyLoraName, a big cat with mountains in the background", "additional_parameters": "", "vae_batch_size": 0, "min_snr_gamma": 5 }

[TingTingin]

You need to increase noise offset the default is 0.1 you have it at .06 or use multires noise Stable Diffusion by default cant do pure white or black or make much change to the dynamic range of an image it's a known issue

[synapsestorm]

You need to increase noise offset the default is 0.1 you have it at .06 or use multires noise Stable Diffusion by default cant do pure white or black or make much change to the dynamic range of an image it's a known issue

Thanks for your reply. I tried 0, and that seemed to work a bit better. Do you know if there's anything else I can try?

[TingTingin]

You need to increase it not decrease it

[synapsestorm]

I tried 0.1, 0.2, 0.5 and 1, and they didn’t fix the issue. How much should I go up by each time? Thanks

[TingTingin]

You can try multires noise instead by doing multires noise iterations at 8 and discount at 0.3 also train for longer and do you have any example inputs and outputs?

[synapsestorm]

Thanks. I tried --multires_noise_iterations="8" (and many other variations) in the "Additional parameters" field, but keep getting an unrecognised args error. How do I input the command properly? I'll share some inputs and outputs after I try using multires noise

[TingTingin]

You can check here https://github.com/kohya-ss/sd-scripts/pull/471#issue-1691791431

[synapsestorm]

Thanks, had to update kohya (doh) - anyways it's still introducing colour to the images. I've tried all kinds of settings combinations but still the best I can manage is: The style is fine, the added colour is not. As training goes on, the colour becomes more pronounced.

And the inputs are all of this style

Thanks :)

[TingTingin]

Do you have any tags that are describing the color i.e sepia line drawing Manga comic etc? If so you'd want to remove those tags so that the model learns the style as a unique idea rather than a combination of those tags

[TingTingin]

You could also try traning without tags to force the stly more or use caption dropout at a high amount like 0.5 this will drop tags 50 percent of the time while training

[synapsestorm]

Nah nothing like that - an example of a tag would be this: "fantasyclassics, weapon, sword, armor, blood, helmet, snake, death, oversized animal, corpse"

I tried without captions and it didn't make a difference, tried with caption dropout at 0.5 which didn't make too much of a difference either.

Not too sure what I can try from here

[TingTingin]

Not sure about your use case or inference environment but you could use controlnet to limit the color pallete at inference time https://github.com/Mikubill/SD-webui-controlnet using t2iadapter_color_sd14v1.pth