search

cswry / SeeSR

[CVPR2024] SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution
Apache License 2.0
438 stars 29 forks source link

About the pretrained weights loading problem in training #67

Open llstela opened 2 months ago

llstela commented 2 months ago

While loading UNet2DConditionModel from pretrained SD-2, ValueError appears, as parameter weights related to image_attentions are missed.

Traceback (most recent call last): File "train_seesr.py", line 680, in unet = UNet2DConditionModel.from_pretrained( File "/opt/conda/envs/seesr/lib/python3.8/site-packages/diffusers/models/modeling_utils.py", line 646, in from_pretrained raise ValueError( ValueError: Cannot load <class 'models.unet_2d_condition.UNet2DConditionModel'> from preset/models/stable-diffusion-2-base because the following keys are missing: down_blocks.1.image_attentions.0.transformer_blocks.0.ff.net.2.bias, up_blocks.3.image_attentions.0.norm.weight, down_blocks.1.image_attentions.1.transformer_blocks.0.attn1.to_out.0.weight, ...

Zebraside commented 2 months ago

I have the same problem. It seems that the problem arises when use_image_cross_attention=True in UNet2DConditionModel.from_pretrained. It is not used in some inference scripts.

Can someone clarify is it a mandatory setting for training?

llstela commented 2 months ago

I have the same problem. It seems that the problem arises when use_image_cross_attention=True in UNet2DConditionModel.from_pretrained. It is not used in some inference scripts.

Can someone clarify is it a mandatory setting for training?

I tried to overwrite the "UNet2DConditionModel.from_pretrained" function. It can work, but I'm not sure it's the right way to solve this problem.

Renzhihan commented 2 months ago

May I ask if you get a mismatch between the output SR image and the input LR image at the beginning of the training? Meanwhile, I specified the use_image_cross_attention=True, --pretrained_model_name_or_path, and --unet_model_name_or_path, but I didn't catch the error. @llstela @Zebraside

Zebraside commented 1 month ago

The error occurs when only --pretrained_model_name_or_path is set. (# resume from pretrained SD branch). I'm using stable-diffusion-2-base from HuggingFace as base model.