Weird result when training a 2-1 768 LoRA

[Bammargiela]

I'm trying to train a style that has some very fine details. I've tried a lot of different methods to get the details and 768 seems to be my best bet. I'm running into a problem where the faces are strange and uncanny. I will leave an example and configuration below. I've tried both finetune and dreambooth because I thought it might have been finetune specific. It did the same thing with dreambooth. To be clear I get normal looking people when trained at 512x512 with the same configuration so I don't believe its the dataset. The first image is at the 22nd epoch. I could train for longer but the issue wasn't seeming to resolve itself and on 512 it looked fine within the first few epochs.

[model_arguments] v2 = true v_parameterization = true pretrained_model_name_or_path = "/content/pretrained_model/stable-diffusion-2-1-768v.safetensors"

[additional_network_arguments] no_metadata = false unet_lr = 1.0 text_encoder_lr = 0.5 network_module = "lycoris.kohya" network_dim = 40 network_alpha = 40 network_args = [ "conv_dim=40", "conv_alpha=40", "algo=lora",] network_train_unet_only = false network_train_text_encoder_only = false

[optimizer_arguments] optimizer_type = "DAdaptation" learning_rate = 1.0 max_grad_norm = 1.0 lr_scheduler = "cosine" lr_warmup_steps = 10

[dataset_arguments] cache_latents = true debug_dataset = false

[training_arguments] output_dir = "/content/drive/MyDrive/LoRA/output" output_name = "olereliable2-1_dreambooth" save_precision = "fp16" save_every_n_epochs = 1 train_batch_size = 6 max_token_length = 225 mem_eff_attn = false xformers = true max_train_epochs = 30 max_data_loader_n_workers = 8 persistent_data_loader_workers = true gradient_checkpointing = false gradient_accumulation_steps = 1 mixed_precision = "fp16" logging_dir = "/content/LoRA/logs" log_prefix = "olereliable2-1_dreambooth" lowram = true

[sample_prompt_arguments] sample_every_n_epochs = 1 sample_sampler = "euler_a"

[dreambooth_arguments] prior_loss_weight = 1.0

[saving_arguments] save_model_as = "safetensors"

[Linaqruf]

Hi, probably this is because of using DAdaptation, we don't know what is the best config for DAdaptation, or atleast me.

[Bammargiela]

Ok thanks that may be it. I got the settings from a YouTube video. I will keep playing around with different settings.