issue about text_encoder

[aihaozi]

Hello, I encountered bugs when I run the code. "text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device)" something like that: ..size mismatch for text_model.encoder.layers.11.mlp.fc2.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for text_model.encoder.layers.11.layer_norm2.weight: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for text_model.encoder.layers.11.layer_norm2.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for text_model.final_layer_norm.weight: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for text_model.final_layer_norm.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]).

Could you help me fix it?

[aihaozi]

@ogkalu2 Hello, I met another bug: AttributeError: type object 'DDPMScheduler' has no attribute 'from_pretrained'. What is the version of transformers and diffusers you used?

[ogkalu2]

For the first, replace "openai/clip-vit-large-patch14" with args.unet for both the tokenizer and text encoder and see if that works. Looks like you're trying to train on v2 or 2.1 correct ?

For the second, pick another scheduler or replace .from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="scheduler") with (beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000). I used PNDMScheduler as that's the default for SD. Although i'm not sure why you're having that error.

[aihaozi]

Hi, what is the version of transformers and diffusers package you used?

[ogkalu2]

I used the latest versions of both. Just pip install diffusers/transformers. Trained on a vast.ai instance