Stable-diffusion-2-1, the config of text_encoder does not match

[mrluin]

Hello,

I tried used stable-diffusion-2-1, i load sd and open_clip checkpoint from hugging face, where sd-2-1(https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main), and open_clip vit-h-14 (https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/tree/main).

But I see that the configs of text_encoder from sd and open_clip don't match: the num_hidden_layer of open_clip model is 24, and num_hidden_layer of sd model is 23, what wrong with it?

[zhuqiangLu]

I find the same problem as well, I tried initialize a diffusers pipe with CLIP-ViT-H-14 text encoder and will never get a good result, it looks like the text encoder used in stable diffusion 2 is a private one.

[QingyangZhang]

CLIPTextConfig { "_name_or_path": "openai/clip-vit-large-patch14", "attention_dropout": 0.0, "bos_token_id": 0, "dropout": 0.0, "eos_token_id": 2, "hidden_act": "quick_gelu", "hidden_size": 768, "initializer_factor": 1.0, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-05, "max_position_embeddings": 77, "model_type": "clip_text_model", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 1, "projection_dim": 768, "transformers_version": "4.28.1", "vocab_size": 49408 }

CLIPTextConfig { "_name_or_path": "/root/.cache/huggingface/hub/models--stabilityai--stable-diffusion-2-1/snapshots/5cae40e6a2745ae2b01ad92ae5043f95f23644d6/text_encoder", "architectures": [ "CLIPTextModel" ], "attention_dropout": 0.0, "bos_token_id": 0, "dropout": 0.0, "eos_token_id": 2, "hidden_act": "gelu", "hidden_size": 1024, "initializer_factor": 1.0, "initializer_range": 0.02, "intermediate_size": 4096, "layer_norm_eps": 1e-05, "max_position_embeddings": 77, "model_type": "clip_text_model", "num_attention_heads": 16, "num_hidden_layers": 23, "pad_token_id": 1, "projection_dim": 512, "torch_dtype": "float32", "transformers_version": "4.28.1", "vocab_size": 49408 }

I try to print out the config. It seems that they are different.