[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
What happened?
A while back I've reported an issue where I couldn't extract/convert trained checkpoint in DB ext for further training. After few updates I can extract it but I'm unable to use it to train it further.
Steps to reproduce the problem
Create new model using previously trained ckpt as base
Try to train even with default settings
Commit and libraries
venv "G:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
Version: v1.8.0
Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5
Installing xformers
Installing requirements
If submitting an issue on github, please provide the full startup log for debugging purposes.
Initializing Dreambooth
Dreambooth revision: 45a12fe5950bf93205b6ef2b7511eb94052a241f
Checking xformers...
Checking bitsandbytes...
Checking bitsandbytes (ALL!)
Checking Dreambooth requirements...
Installed version of bitsandbytes: 0.43.0
[Dreambooth] bitsandbytes v0.43.0 is already installed.
Installed version of accelerate: 0.21.0
[Dreambooth] accelerate v0.21.0 is already installed.
Installed version of dadaptation: 3.2
[Dreambooth] dadaptation v3.2 is already installed.
Installed version of diffusers: 0.27.2
[Dreambooth] diffusers v0.25.0 is already installed.
Installed version of discord-webhook: 1.3.0
[Dreambooth] discord-webhook v1.3.0 is already installed.
Installed version of fastapi: 0.94.0
[Dreambooth] fastapi is already installed.
Installed version of gitpython: 3.1.32
[Dreambooth] gitpython v3.1.40 is not installed.
Successfully installed gitpython-3.1.43
Installed version of pytorch_optimizer: 2.12.0
[Dreambooth] pytorch_optimizer v2.12.0 is already installed.
Installed version of Pillow: 9.5.0
[Dreambooth] Pillow is already installed.
Installed version of tqdm: 4.66.2
[Dreambooth] tqdm is already installed.
Installed version of tomesd: 0.1.3
[Dreambooth] tomesd v0.1.2 is already installed.
Installed version of tensorboard: 2.16.2
[Dreambooth] tensorboard v2.13.0 is not installed.
Successfully installed tensorboard-2.13.0
[+] torch version 2.1.2+cu121 installed.
[+] torchvision version 0.16.2+cu121 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.27.2 installed.
[+] bitsandbytes version 0.43.0 installed.
[+] xformers version 0.0.23.post1 installed.
loading WD14-tagger reqs from G:\stable-diffusion-webui\extensions\stable-diffusion-webui-wd14-tagger\requirements.txt
Checking WD14-tagger requirements.
Launching Web UI with arguments: --reinstall-xformers --xformers --vae-dir G:\AI\VAE --ckpt-dir G:\AI\SD_Models --lora-dir G:\AI\Lora --embeddings-dir G:\AI\embeddings --esrgan-models-path G:\AI\ESRGAN
2024-04-07 20:00:46.266764: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-04-07 20:00:46.814290: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
== WD14 tagger /gpu:0, uname_result(system='Windows', node='AsusX670E', release='10', version='10.0.22631', machine='AMD64') ==
Loading weights [b4f887259f] from G:\AI\SD_Models\HD6.fp16.safetensors
Creating model from config: G:\AI\SD_Models\HD6.fp16.yaml
Running on local URL: http://127.0.0.1:7860
Command Line Arguments
--xformers,
Console logs
Extracting config from G:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\..\configs\v1-training-unfrozen.yaml
Extracting checkpoint from G:\AI\SD_Models\HD8.fp16.safetensors
Duration: 00:00:07
Updating scheduler name to: DDIM
Initializing dreambooth training...
WARNING:dreambooth.train_dreambooth:Wandb API key not set. Please set WANDB_API_KEY environment variable to use wandb.
0 cached latents
0it [00:00, ?it/s] Traceback (most recent call last):████████████████████▌ | 2/5 [00:00<00:00, 50.63it/s]
File "G:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 735, in start_training
result = main(class_gen_method=class_gen_method)
File "G:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 2003, in main
return inner_loop()
File "G:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 126, in decorator
return function(batch_size, grad_size, prof, *args, **kwargs)
File "G:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 449, in inner_loop
text_encoder = text_encoder_cls.from_pretrained(
File "G:\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py", line 2881, in from_pretrained
) = cls._load_pretrained_model(
File "G:\stable-diffusion-webui\venv\lib\site-packages\transformers\modeling_utils.py", line 3278, in _load_pretrained_model
raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
While copying the parameter named "text_model.embeddings.token_embedding.weight", whose dimensions in the model are torch.Size([49408, 768]) and whose dimensions in the checkpoint are torch.Size([49408, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.embeddings.position_embedding.weight", whose dimensions in the model are torch.Size([77, 768]) and whose dimensions in the checkpoint are torch.Size([77, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.0.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.1.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.2.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.3.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.4.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.5.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.6.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.7.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.8.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.9.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.10.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.k_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.k_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.v_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.v_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.q_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.q_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.out_proj.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.self_attn.out_proj.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.layer_norm1.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.layer_norm1.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.mlp.fc1.weight", whose dimensions in the model are torch.Size([3072, 768]) and whose dimensions in the checkpoint are torch.Size([3072, 768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.mlp.fc1.bias", whose dimensions in the model are torch.Size([3072]) and whose dimensions in the checkpoint are torch.Size([3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.mlp.fc2.weight", whose dimensions in the model are torch.Size([768, 3072]) and whose dimensions in the checkpoint are torch.Size([768, 3072]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.mlp.fc2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.layer_norm2.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.encoder.layers.11.layer_norm2.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.final_layer_norm.weight", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
While copying the parameter named "text_model.final_layer_norm.bias", whose dimensions in the model are torch.Size([768]) and whose dimensions in the checkpoint are torch.Size([768]), an exception occurred : ('Cannot copy out of meta tensor; no data!',).
Loading text encoder...: 40%|███████████████████████▌ | 2/5 [00:00<00:00, 3.38it/s]
Duration: 00:00:01
Duration: 00:00:02
Is there an existing issue for this?
What happened?
A while back I've reported an issue where I couldn't extract/convert trained checkpoint in DB ext for further training. After few updates I can extract it but I'm unable to use it to train it further.
Steps to reproduce the problem
Commit and libraries
venv "G:\stable-diffusion-webui\venv\Scripts\Python.exe" Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] Version: v1.8.0 Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5 Installing xformers Installing requirements If submitting an issue on github, please provide the full startup log for debugging purposes.
Initializing Dreambooth Dreambooth revision: 45a12fe5950bf93205b6ef2b7511eb94052a241f Checking xformers... Checking bitsandbytes... Checking bitsandbytes (ALL!) Checking Dreambooth requirements... Installed version of bitsandbytes: 0.43.0 [Dreambooth] bitsandbytes v0.43.0 is already installed. Installed version of accelerate: 0.21.0 [Dreambooth] accelerate v0.21.0 is already installed. Installed version of dadaptation: 3.2 [Dreambooth] dadaptation v3.2 is already installed. Installed version of diffusers: 0.27.2 [Dreambooth] diffusers v0.25.0 is already installed. Installed version of discord-webhook: 1.3.0 [Dreambooth] discord-webhook v1.3.0 is already installed. Installed version of fastapi: 0.94.0 [Dreambooth] fastapi is already installed. Installed version of gitpython: 3.1.32 [Dreambooth] gitpython v3.1.40 is not installed. Successfully installed gitpython-3.1.43
Installed version of pytorch_optimizer: 2.12.0 [Dreambooth] pytorch_optimizer v2.12.0 is already installed. Installed version of Pillow: 9.5.0 [Dreambooth] Pillow is already installed. Installed version of tqdm: 4.66.2 [Dreambooth] tqdm is already installed. Installed version of tomesd: 0.1.3 [Dreambooth] tomesd v0.1.2 is already installed. Installed version of tensorboard: 2.16.2 [Dreambooth] tensorboard v2.13.0 is not installed. Successfully installed tensorboard-2.13.0
[+] torch version 2.1.2+cu121 installed. [+] torchvision version 0.16.2+cu121 installed. [+] accelerate version 0.21.0 installed. [+] diffusers version 0.27.2 installed. [+] bitsandbytes version 0.43.0 installed. [+] xformers version 0.0.23.post1 installed.
loading WD14-tagger reqs from G:\stable-diffusion-webui\extensions\stable-diffusion-webui-wd14-tagger\requirements.txt Checking WD14-tagger requirements. Launching Web UI with arguments: --reinstall-xformers --xformers --vae-dir G:\AI\VAE --ckpt-dir G:\AI\SD_Models --lora-dir G:\AI\Lora --embeddings-dir G:\AI\embeddings --esrgan-models-path G:\AI\ESRGAN 2024-04-07 20:00:46.266764: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0
. 2024-04-07 20:00:46.814290: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variableTF_ENABLE_ONEDNN_OPTS=0
. == WD14 tagger /gpu:0, uname_result(system='Windows', node='AsusX670E', release='10', version='10.0.22631', machine='AMD64') == Loading weights [b4f887259f] from G:\AI\SD_Models\HD6.fp16.safetensors Creating model from config: G:\AI\SD_Models\HD6.fp16.yaml Running on local URL: http://127.0.0.1:7860Command Line Arguments
Console logs
Additional information
No response