search

bmaltais / kohya_ss

Apache License 2.0
9.71k stars 1.25k forks source link

(Flux) Dreambooth trained model generates noise #2939

Open antey3064 opened 3 weeks ago

antey3064 commented 3 weeks ago

What could be the reason?

Guidance Scale:1

last7_e000001_00_20241031021302_54

02:07:45-358115 INFO Start training Dreambooth... 02:07:45-359117 INFO Validating lr scheduler arguments... 02:07:45-360117 INFO Validating optimizer arguments... 02:07:45-362120 INFO Validating D:/SD/MyImages/testPink\log existence and writability... SUCCESS 02:07:45-363120 INFO Validating D:/SD/MyImages/testPink\model existence and writability... SUCCESS 02:07:45-363120 INFO Validating D:/SD/webui_forge_cu121_torch231/webui/models/Stable-diffusion/flux1-dev.safetensors existence... SUCCESS 02:07:45-365122 INFO Validating D:/SD/MyImages/testPink\img existence... SUCCESS 02:07:45-366122 INFO Folder 1_xyzpinkdress dress: 1 repeats found 02:07:45-367123 INFO Folder 1_xyzpinkdress dress: 8 images found 02:07:45-368124 INFO Folder 1_xyzpinkdress dress: 8 1 = 8 steps 02:07:45-369125 INFO Regularization factor: 1 02:07:45-370126 INFO Total steps: 8 02:07:45-370126 INFO Train batch size: 1 02:07:45-373129 INFO Gradient accumulation steps: 1 02:07:45-375797 INFO Epoch: 18 02:07:45-377055 INFO max_train_steps (8 / 1 / 1 18 * 1) = 144 02:07:45-378067 INFO lr_warmup_steps = 0 02:07:45-382736 INFO Saving training config to D:/SD/MyImages/testPink\model\last7_20241031-020745.json... 02:07:45-384965 INFO Executing command: D:\SD\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1 --num_machines 1 --num_cpu_threads_per_process 2 D:/SD/kohya_ss/sd-scripts/flux_train.py --config_file D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745.toml D:\SD\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( D:\SD\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( 2024-10-31 02:07:56 INFO Loading settings from train_util.py:4435 D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745.toml... INFO D:/SD/MyImages/testPink\model/config_dreambooth-20241031-020745 train_util.py:4454 2024-10-31 02:07:56 INFO Using DreamBooth method. flux_train.py:107 INFO prepare images. train_util.py:1956 INFO get image size from name of cache files train_util.py:1873 100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s] INFO set image size from cache files: 8/8 train_util.py:1901 INFO found directory D:\SD\MyImages\testPink\img\1_xyzpinkdress dress train_util.py:1903 contains 8 image files read caption: 100%|██████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s] INFO 8 train images with repeating. train_util.py:1997 INFO 0 reg images. train_util.py:2000 WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:2005 INFO [Dataset 0] config_util.py:567 batch_size: 1 resolution: (1024, 1024) enable_bucket: True network_multiplier: 1.0 min_bucket_reso: 256 max_bucket_reso: 2048 bucket_reso_steps: 64 bucket_no_upscale: True

                           [Subset 0 of Dataset 0]
                             image_dir: "D:\SD\MyImages\testPink\img\1_xyzpinkdress dress"
                             image_count: 8
                             num_repeats: 1
                             shuffle_caption: False
                             keep_tokens: 0
                             keep_tokens_separator:
                             caption_separator: ,
                             secondary_separator: None
                             enable_wildcard: False
                             caption_dropout_rate: 0
                             caption_dropout_every_n_epoches: 0
                             caption_tag_dropout_rate: 0.0
                             caption_prefix: None
                             caption_suffix: None
                             color_aug: False
                             flip_aug: False
                             face_crop_aug_range: None
                             random_crop: False
                             token_warmup_min: 1
                             token_warmup_step: 0
                             alpha_mask: False
                             custom_attributes: {}
                             is_reg: False
                             class_tokens: xyzpinkdress dress
                             caption_extension: .txt

                INFO     [Dataset 0]                                                              config_util.py:573
                INFO     loading image sizes.                                                      train_util.py:923

100%|████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<?, ?it/s] INFO make buckets train_util.py:946 WARNING min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is train_util.py:963 set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計 算されるため、min_bucket_resoとmax_bucket_resoは無視されます INFO number of images (including repeats) / train_util.py:992 各bucketの画像枚数（繰り返し回数を含む） INFO bucket 0: resolution (896, 1088), count: 6 train_util.py:997 INFO bucket 1: resolution (1024, 1024), count: 2 train_util.py:997 INFO mean ar error (without repeats): 0.01759017994531699 train_util.py:1002 INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:62 INFO prepare accelerator flux_train.py:177 accelerator device: cuda INFO Building AutoEncoder flux_utils.py:152 INFO Loading state dict from flux_utils.py:157 D:/SD/webui_forge_cu121_torch231/webui/models/VAE/ae.safetensors INFO Loaded AE: flux_utils.py:160 2024-10-31 02:07:57 INFO [Dataset 0] train_util.py:2480 INFO caching latents with caching strategy. train_util.py:1048 INFO caching latents... train_util.py:1093 100%|██████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 1998.24it/s] D:\SD\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn( You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 2024-10-31 02:07:58 INFO Building CLIP flux_utils.py:165 INFO Loading state dict from flux_utils.py:258 D:/SD/webui_forge_cu121_torch231/webui/models/text_encoder/clip_l.safeten sors INFO Loaded CLIP: flux_utils.py:261 INFO Loading state dict from flux_utils.py:306 D:/SD/webui_forge_cu121_torch231/webui/models/text_encoder/t5xxl_fp16.saf etensors INFO Loaded T5xxl: flux_utils.py:309 2024-10-31 02:08:07 INFO [Dataset 0] train_util.py:2502 INFO caching Text Encoder outputs with caching strategy. train_util.py:1227 INFO checking cache validity... train_util.py:1238 100%|███████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 584.69it/s] INFO no Text Encoder outputs to cache train_util.py:1265 INFO cache Text Encoder outputs for sample prompt: flux_train.py:240 D:/SD/MyImages/testPink\model\sample/prompt.txt INFO cache Text Encoder outputs for prompt: woman standing, dress, flux_train.py:250 xyzpinkdress, full body 2024-10-31 02:08:08 INFO cache Text Encoder outputs for prompt: flux_train.py:250 2024-10-31 02:08:09 INFO Checking the state dict: Diffusers or BFL, dev or schnell flux_utils.py:62 INFO Building Flux model dev from BFL checkpoint flux_utils.py:116 INFO Loading state dict from flux_utils.py:133 D:/SD/webui_forge_cu121_torch231/webui/models/Stable-diffusion/flux1-dev. safetensors INFO Loaded Flux: flux_utils.py:145 FLUX: Gradient checkpointing enabled. CPU offload: False INFO enable block swap: blocks_to_swap=10 flux_train.py:295 number of trainable parameters: 11901408320 prepare optimizer, data loader etc. INFO use Adafactor optimizer | {'relative_step': True} train_util.py:4748 INFO relative_step is true / relative_stepがtrueです train_util.py:4751 WARNING learning rate is used as initial_lr / 指定したlearning train_util.py:4753 rateはinitial_lrとして使用されます WARNING unet_lr and text_encoder_lr are ignored / train_util.py:4765 unet_lrとtext_encoder_lrは無視されます INFO use adafactor_scheduler / スケジューラにadafactor_schedulerを使用します train_util.py:4770 running training / 学習開始 num examples / サンプル数: 8 num batches per epoch / 1epochのバッチ数: 8 num epochs / epoch数: 18 batch size per device / バッチサイズ: 1 gradient accumulation steps / 勾配を合計するステップ数 = 1 total optimization steps / 学習ステップ数: 144 steps: 0%| | 0/144 [00:00<?, ?it/s] epoch 1/18 2024-10-31 02:08:22 INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:715 D:\SD\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.) return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass steps: 6%|███▍ | 8/144 [02:15<38:23, 16.94s/it, avr_loss=1.8]2024-10-31 02:10:25 INFO flux_train_utils.py:59 INFO generating sample images at step / サンプル画像生成 ステップ: 8 flux_train_utils.py:60 INFO prompt: woman standing, dress, xyzpinkdress, full body flux_train_utils.py:176 INFO height: 1024 flux_train_utils.py:178 INFO width: 1024 flux_train_utils.py:179 INFO sample_steps: 20 flux_train_utils.py:180 INFO scale: 1.0 flux_train_utils.py:181 INFO seed: 54 flux_train_utils.py:184 Using cached text encoder outputs for prompt: woman standing, dress, xyzpinkdress, full body Encoding prompt: woman standing, dress, xyzpinkdress, full body 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [02:31<00:00, 7.56s/it] 100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [02:31<00:00, 7.74s/it] epoch 2/18

antey3064 commented 3 weeks ago

problem solved. The problem was in the incorrectly configured Dreambooth

kubakubkub commented 3 weeks ago

problem solved. The problem was in the incorrectly configured Dreambooth

You might sharing your config ?