SDXL error

[tin2tin]

C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\train_dreambooth_lora_sdxl.py:444: UserWarning: You need not use --class_prompt without --with_prior_preservation.
  warnings.warn("You need not use --class_prompt without --with_prior_preservation.")
C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\venv\lib\site-packages\accelerate\accelerator.py:382: UserWarning: `log_with=tensorboard` was passed but no supported trackers are currently installed.
  warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
10/22/2023 21:21:52 - INFO - __main__ - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
{'dynamic_thresholding_ratio', 'thresholding', 'variance_type', 'clip_sample_range'} was not found in config. Values will be initialized to default values.
Traceback (most recent call last):
  File "C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\train_dreambooth_lora_sdxl.py", line 1449, in <module>
    main(args)
  File "C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\train_dreambooth_lora_sdxl.py", line 721, in main
    text_encoder_two = text_encoder_cls_two.from_pretrained(
  File "C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\venv\lib\site-packages\transformers\modeling_utils.py", line 2864, in from_pretrained
    raise EnvironmentError(
OSError: Error no file named pytorch_model.bin, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory C:\Users\45239\Documents\LORA\simple-lora-dreambooth-trainer-main\base-model-path-sdxl\sdxl_fp16_only.
['C:\\Users\\45239\\Documents\\LORA\\simple-lora-dreambooth-trainer-main\\venv\\Scripts\\python.exe', 'C:\\Users\\45239\\Documents\\LORA\\simple-lora-dreambooth-trainer-main\\train_dreambooth_lora_sdxl.py', '--pretrained_model_name_or_path', 'C:\\Users\\45239\\Documents\\LORA\\simple-lora-dreambooth-trainer-main\\base-model-path-sdxl\\sdxl_fp16_only', '--instance_data_dir', 'C:/Users/45239/Documents/LORA/Werner Img XL', '--instance_prompt', 'WernerHerzog', '--class_prompt', 'WernerHerzog', '--output_dir', 'C:/Users/45239/Documents/LORA/Output XL', '--resolution', '1024', '--train_batch_size', '1', '--num_train_epochs', '32', '--gradient_accumulation_steps', '1', '--learning_rate', '0.0001', '--optimizer', 'AdamW', '--lr_scheduler', 'constant', '--lr_warmup_steps', '0', '--mixed_precision', 'fp16', '--prior_generation_precision', 'fp16', '--rank', '4', '--pre_compute_text_embeddings', '--gradient_checkpointing', '--enable_xformers_memory_efficient_attention']

[johnman3032]

I am not able to replicate the issue on my end, although I suspect it may be because I forgot to list "GIT" as a requirement. Check out the updated README for more info. Installing GIT(bash) can be a bit complicated because it asks for a lot of options, but unfortunately there is no better way to download large model files from Huggingface.

[tin2tin]

I actually already had GIT(bash) installed, but I tried updating it. However, the error is still there. And training on 1.5 already works.

I'm working on this free genAI add-on for Blender: https://github.com/tin2tin/Pallaidium Last night, I've implemented support for using multiple LoRAs, however the Diffusers solution(which I use for the implementation), only supports SDXL LoRAs. So, I would love to get SDXL training working. :-)

Btw. have you seen this new, faster and smaller version of SDXL? https://huggingface.co/segmind/SSD-1B Maybe this could be used for training SDXL LoRAs on 6 GB of VRAM?

[tin2tin]

I noticed that my SD 1.5 - which works for training - has the missing files added as symlinks:

The SD XL folder doesn't have these and fail?

[johnman3032]

It is definitely possible to train on https://huggingface.co/segmind/SSD-1B, but I don't know whether or not it gives benefits in LoRA training. You would have to take the "diffusion_pytorch_model.bin" and delete the other one. But that leaves you with the full-precision fp32 model, which may be very VRAM costly. Maybe it is better to leave the "diffusion_pytorch_model.fp16.bin" and delete the other one, while re-naming "diffusion_pytorch_model.fp16.bin" to "diffusion_pytorch_model.bin", because the Python script expects a "diffusion_pytorch_model.bin" or "diffusion_pytorch_model.safetensors".

Regarding your error, you may want to check what you are putting into "Diffusers Base Model Path", maybe just use "base-model-path-sdxl/sdxl_fp16_only", or use the "Browse" button to point to your "sdxl_fp16_only" file directly.