SDXL1.0 lora training error

[Jon-Zbw]

File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/library/sdxl_original_unet.py", line 1080, in forward h = torch.cat([h, hs.pop()], dim=1) RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 58 but got size 57 for tensor number 1 in the list.

[Jon-Zbw]

whole message: /home/bowen/zhang3/LDM/lora-scripts/sd-scripts/venv/lib/python3.9/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn("None of the inputs have requires_grad=True. Gradients will be None") Traceback (most recent call last): File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/./sdxl_train_network.py", line 174, in trainer.train(args) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/train_network.py", line 775, in train noise_pred = self.call_unet( File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/./sdxl_train_network.py", line 154, in call_unet noise_pred = unet(noisy_latents, timesteps, text_embedding, vector_embedding) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/venv/lib/python3.9/site-packages/accelerate/utils/operations.py", line 521, in forward return model_forward(*args, *kwargs) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/venv/lib/python3.9/site-packages/accelerate/utils/operations.py", line 509, in call return convert_to_fp32(self.model_forward(args, kwargs)) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/venv/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast return func(*args, **kwargs) File "/home/bowen/zhang3/LDM/lora-scripts/sd-scripts/library/sdxl_original_unet.py", line 1080, in forward h = torch.cat([h, hs.pop()], dim=1) RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 58 but got size 57 for tensor number 1 in the list. steps: 0%|

[kohya-ss]

Unfortunately, the bucket_reso_steps lower than 32 is not supported in SDXL. Please use 32 or higher.

I will add checking for it.

[Jon-Zbw]

@kohya-ss thks for replying, the problem is found to be set : bucket_no_upscale=true

[kohya-ss]

Even if bucket_no_upscale=true, the resolution of the bucket should be divisible by bucket_reso_steps. So if you set bucket_reso_steps to 32 or higher, and get the error, there may be a bug. What the value you use for bucket_reso_steps option?

[Jon-Zbw]

@kohya-ss Unfortunately, it's 32 ，But the situation at that time was：min_bucket_reso=256 max_bucket_reso=1024 ，about traindata ， both width and height are greater than 1024

[Jon-Zbw]

after got error ,i setted min_bucket_reso=640 max_bucket_reso=1024 resolution = "640,1024" bucket_no_upscale=false and resize the traindata to 640*1024，then it worked