search

Francis-Rings / MotionEditor

[CVPR2024] MotionEditor is the first diffusion-based model capable of video motion editing.
106 stars 6 forks source link

Guidance on Checkpoint Preparation Needed #2

Open DAVEISHAN opened 1 month ago

DAVEISHAN commented 1 month ago

Thank you for your excellent work and commitment to open source!

I am attempting to execute the training script using the original settings. Could you provide additional details regarding the setup of the checkpoint folder as described in this section of the README?

Initially, I tried running the script using the command: python train_bg.py --config="configs/case-1/train-bg.yaml". To prepare the checkpoints, I attempted to load the pre-trained weights directly from the Hugging Face repository ID: runwayml/stable-diffusion-v1-5 by updating the pretrained_model_path in train-bg.yaml. This approach successfully loaded the weights for the tokenizer, text_encoder, vae, and noise_scheduler within train_bg.py. However, when loading the unet weights, I encountered an error message regarding 'missing keys during the loading process' (see the following message). I am unsure if this is an expected behavior or if there may be an issue with how the weights are being loads from runwayml/stable-diffusion-v1-5.

The message while loading unet: Some weights of UNet2DConditionModel were not initialized from the model checkpoint at runwayml/stable-diffusion-v1-5 and are newly initialized: ['controlnet_adapter.body.8.norm_self_temp.weight', 'controlnet_adapter.body.6.ff.net.0.proj.weight', 'controlnet_adapter.body.0.attn_self_temp.to_k.weight', 'up_blocks.1.resnets.2.temp_conv1.bias', 'controlnet_adapter.body.3.block2.bias', 'controlnet_adapter.body.9.ff.net.0.proj.bias', 'controlnet_adapter.body.1.attn_pose.to_out.0.weight', 'controlnet_adapter.body.1.attn_pose.to_k.weight', 'controlnet_adapter.body.6.norm_temp.bias', 'controlnet_adapter.body.7.attn_pose.to_k.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'up_blocks.1.attentions.2.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.9.attn_self_temp.to_q.weight', 'controlnet_adapter.body.11.block1.bias', 'controlnet_adapter.body.5.attn_temp.to_q.weight', 'controlnet_adapter.body.10.ff.net.0.proj.bias', 'up_blocks.3.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.6.ff_norm.bias', 'controlnet_adapter.body.11.block1.weight', 'controlnet_adapter.body.8.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.0.attn_pose.to_v.weight', 'controlnet_adapter.body.5.attn_self_temp.to_v.weight', 'controlnet_adapter.body.3.norm_temp.weight', 'controlnet_adapter.body.2.attn_pose.to_k.weight', 'up_blocks.2.attentions.2.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.11.ff.net.0.proj.bias', 'controlnet_adapter.body.4.attn_pose.to_k.weight', 'controlnet_adapter.body.6.cross_pose_norm.bias', 'controlnet_adapter.body.3.attn_temp.to_k.weight', 'controlnet_adapter.body.0.ff.net.2.weight', 'controlnet_adapter.body.2.attn_pose.to_out.0.weight', 'controlnet_adapter.body.1.norm_self_temp.bias', 'controlnet_adapter.body.0.ff_norm.weight', 'mid_block.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.8.block1.weight', 'up_blocks.0.resnets.1.temp_conv2.bias', 'down_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.9.attn_temp.to_k.weight', 'controlnet_adapter.body.0.block1.bias', 'controlnet_adapter.body.11.cross_pose_norm.bias', 'controlnet_adapter.body.10.norm_temp.weight', 'down_blocks.2.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.4.attn_pose.to_v.weight', 'down_blocks.0.resnets.0.temp_conv1.bias', 'down_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'down_blocks.3.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.11.ff.net.2.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.0.ff.net.2.bias', 'controlnet_adapter.body.0.cross_pose_norm.weight', 'up_blocks.1.attentions.1.transformer_blocks.0.norm_temp.bias', 'up_blocks.3.resnets.1.temp_conv2.bias', 'down_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'down_blocks.3.resnets.0.temp_conv2.bias', 'controlnet_adapter.body.3.norm_self_temp.weight', 'controlnet_adapter.body.6.ff.net.2.bias', 'controlnet_adapter.body.11.ff.net.2.bias', 'controlnet_adapter.body.10.ff.net.2.bias', 'mid_block.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.9.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.3.attn_pose.to_q.weight', 'up_blocks.3.resnets.2.temp_conv2.bias', 'controlnet_adapter.body.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.1.attn_self_temp.to_out.0.weight', 'up_blocks.3.attentions.1.transformer_blocks.0.norm_temp.weight', 'down_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.0.attn_self_temp.to_v.weight', 'up_blocks.1.attentions.1.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.6.norm_temp.weight', 'down_blocks.3.resnets.0.temp_conv2.weight', 'controlnet_adapter.body.4.ff.net.2.weight', 'up_blocks.3.resnets.0.temp_conv2.bias', 'up_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.2.norm_temp.bias', 'controlnet_adapter.body.5.attn_temp.to_k.weight', 'controlnet_adapter.body.3.block1.weight', 'controlnet_adapter.body.9.attn_temp.to_out.0.weight', 'controlnet_adapter.body.3.norm_self_temp.bias', 'mid_block.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'mid_block.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.1.attn_pose.to_v.weight', 'controlnet_adapter.body.9.block1.weight', 'down_blocks.1.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.7.attn_pose.to_out.0.weight', 'controlnet_adapter.body.11.norm_temp.bias', 'up_blocks.1.resnets.0.temp_conv2.weight', 'down_blocks.0.resnets.0.temp_conv2.bias', 'controlnet_adapter.body.9.cross_pose_norm.weight', 'up_blocks.0.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.0.attn_self_temp.to_q.weight', 'controlnet_adapter.body.7.attn_self_temp.to_k.weight', 'controlnet_adapter.body.6.block1.weight', 'up_blocks.1.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.3.cross_pose_norm.weight', 'down_blocks.1.attentions.1.transformer_blocks.0.norm_temp.bias', 'down_blocks.0.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.2.attn_self_temp.to_k.weight', 'controlnet_adapter.body.4.block1.weight', 'up_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.4.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.6.attn_temp.to_k.weight', 'controlnet_adapter.body.2.attn_temp.to_q.weight', 'controlnet_adapter.body.8.attn_temp.to_v.weight', 'controlnet_adapter.body.3.attn_temp.to_out.0.bias', 'mid_block.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.11.norm_self_temp.bias', 'controlnet_adapter.body.11.attn_pose.to_v.weight', 'controlnet_adapter.body.2.attn_self_temp.to_out.0.bias', 'down_blocks.3.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.11.ff_norm.weight', 'up_blocks.3.resnets.0.temp_conv2.weight', 'down_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'up_blocks.3.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.10.attn_temp.to_v.weight', 'controlnet_adapter.body.1.ff_norm.bias', 'up_blocks.2.attentions.1.transformer_blocks.0.norm_temp.bias', 'up_blocks.2.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.5.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.5.block1.bias', 'controlnet_adapter.body.11.attn_pose.to_k.weight', 'up_blocks.2.attentions.2.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.11.attn_pose.to_out.0.weight', 'up_blocks.1.resnets.2.temp_conv2.weight', 'up_blocks.3.resnets.0.temp_conv1.bias', 'up_blocks.2.attentions.2.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.10.cross_pose_norm.weight', 'up_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.2.attn_temp.to_k.weight', 'controlnet_adapter.body.6.attn_temp.to_out.0.bias', 'up_blocks.2.resnets.0.temp_conv2.weight', 'up_blocks.3.attentions.1.transformer_blocks.0.norm_temp.bias', 'up_blocks.3.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.5.ff.net.2.weight', 'controlnet_adapter.body.8.attn_pose.to_q.weight', 'controlnet_adapter.body.5.attn_self_temp.to_k.weight', 'controlnet_adapter.body.2.cross_pose_norm.weight', 'controlnet_adapter.body.3.ff.net.2.bias', 'controlnet_adapter.body.8.ff_norm.weight', 'controlnet_adapter.body.7.ff.net.0.proj.weight', 'controlnet_adapter.body.1.ff.net.0.proj.weight', 'up_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.8.attn_pose.to_out.0.bias', 'mid_block.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.8.cross_pose_norm.weight', 'down_blocks.0.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'up_blocks.3.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.9.ff_norm.bias', 'controlnet_adapter.body.2.attn_self_temp.to_out.0.weight', 'up_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'mid_block.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.9.attn_temp.to_out.0.bias', 'controlnet_adapter.body.11.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.11.block2.bias', 'controlnet_adapter.body.9.block1.bias', 'controlnet_adapter.body.10.ff_norm.weight', 'up_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.7.norm_self_temp.bias', 'down_blocks.2.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.0.attn_pose.to_k.weight', 'controlnet_adapter.body.1.block1.weight', 'controlnet_adapter.body.2.attn_pose.to_out.0.bias', 'controlnet_adapter.body.8.ff.net.0.proj.weight', 'controlnet_adapter.body.3.attn_pose.to_v.weight', 'controlnet_adapter.body.0.norm_temp.weight', 'up_blocks.1.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.10.block2.bias', 'up_blocks.0.resnets.2.temp_conv1.bias', 'mid_block.resnets.0.temp_conv2.weight', 'up_blocks.2.resnets.2.temp_conv2.bias', 'controlnet_adapter.body.1.attn_self_temp.to_v.weight', 'controlnet_adapter.body.10.attn_temp.to_out.0.bias', 'up_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.11.block2.weight', 'controlnet_adapter.body.3.attn_self_temp.to_k.weight', 'controlnet_adapter.body.10.ff.net.0.proj.weight', 'controlnet_adapter.body.8.ff.net.2.weight', 'controlnet_adapter.body.10.attn_pose.to_v.weight', 'down_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.8.attn_self_temp.to_q.weight', 'controlnet_adapter.body.6.cross_pose_norm.weight', 'down_blocks.2.resnets.1.temp_conv1.weight', 'down_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.4.attn_self_temp.to_v.weight', 'up_blocks.2.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.9.ff.net.2.weight', 'controlnet_adapter.body.5.attn_temp.to_out.0.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.attn_temp.to_k.weight', 'mid_block.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'up_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'mid_block.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.10.norm_self_temp.bias', 'controlnet_adapter.body.1.attn_self_temp.to_k.weight', 'controlnet_adapter.body.11.attn_temp.to_v.weight', 'controlnet_adapter.body.7.block2.weight', 'up_blocks.0.resnets.0.temp_conv2.weight', 'up_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.7.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.5.attn_pose.to_out.0.bias', 'up_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.10.block1.weight', 'down_blocks.2.resnets.1.temp_conv1.bias', 'up_blocks.3.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.9.attn_pose.to_q.weight', 'down_blocks.3.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.5.attn_self_temp.to_q.weight', 'controlnet_adapter.body.4.block1.bias', 'controlnet_adapter.body.5.ff_norm.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.6.block2.weight', 'controlnet_adapter.body.0.attn_self_temp.to_out.0.bias', 'down_blocks.0.resnets.1.temp_conv1.bias', 'down_blocks.1.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.1.ff.net.2.bias', 'controlnet_adapter.body.10.block2.weight', 'controlnet_adapter.body.8.norm_temp.bias', 'controlnet_adapter.body.10.ff_norm.bias', 'controlnet_adapter.body.6.ff.net.2.weight', 'controlnet_adapter.body.4.attn_pose.to_out.0.bias', 'down_blocks.1.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.4.attn_temp.to_v.weight', 'controlnet_adapter.body.8.ff_norm.bias', 'controlnet_adapter.body.0.block2.weight', 'controlnet_adapter.body.1.attn_temp.to_k.weight', 'down_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.10.attn_pose.to_k.weight', 'controlnet_adapter.body.5.attn_pose.to_v.weight', 'controlnet_adapter.body.7.ff_norm.bias', 'controlnet_adapter.body.2.ff_norm.weight', 'controlnet_adapter.body.6.norm_self_temp.bias', 'controlnet_adapter.body.3.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.0.norm_temp.bias', 'up_blocks.3.attentions.2.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.8.attn_pose.to_k.weight', 'up_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'down_blocks.0.attentions.1.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.6.attn_self_temp.to_k.weight', 'down_blocks.0.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'up_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.5.ff_norm.bias', 'up_blocks.1.attentions.2.transformer_blocks.0.norm_temp.bias', 'up_blocks.3.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.6.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.10.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.3.ff_norm.bias', 'controlnet_adapter.body.0.block1.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'down_blocks.1.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.4.attn_temp.to_out.0.weight', 'up_blocks.1.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.7.attn_temp.to_v.weight', 'up_blocks.2.attentions.1.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.1.ff.net.2.weight', 'controlnet_adapter.body.3.attn_self_temp.to_v.weight', 'controlnet_adapter.body.3.attn_pose.to_out.0.bias', 'up_blocks.3.resnets.2.temp_conv1.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.3.block1.bias', 'up_blocks.1.attentions.2.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.4.ff_norm.bias', 'up_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'up_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.11.attn_self_temp.to_q.weight', 'up_blocks.1.attentions.2.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.9.ff.net.0.proj.weight', 'controlnet_adapter.body.7.attn_temp.to_out.0.weight', 'controlnet_adapter.body.10.norm_self_temp.weight', 'up_blocks.2.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.6.norm_self_temp.weight', 'controlnet_adapter.body.1.ff.net.0.proj.bias', 'controlnet_adapter.body.5.norm_temp.bias', 'controlnet_adapter.body.6.attn_self_temp.to_q.weight', 'controlnet_adapter.body.10.attn_temp.to_k.weight', 'controlnet_adapter.body.8.block2.bias', 'up_blocks.0.resnets.2.temp_conv1.weight', 'down_blocks.1.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.2.ff.net.2.bias', 'controlnet_adapter.body.8.attn_temp.to_out.0.weight', 'controlnet_adapter.body.0.ff.net.0.proj.weight', 'controlnet_adapter.body.8.attn_temp.to_q.weight', 'controlnet_adapter.body.4.attn_temp.to_q.weight', 'controlnet_adapter.body.1.attn_temp.to_q.weight', 'controlnet_adapter.body.7.ff.net.2.bias', 'controlnet_adapter.body.10.attn_temp.to_out.0.weight', 'controlnet_adapter.body.1.block1.bias', 'controlnet_adapter.body.2.norm_temp.weight', 'controlnet_adapter.body.10.cross_pose_norm.bias', 'controlnet_adapter.body.0.attn_pose.to_out.0.weight', 'controlnet_adapter.body.5.block2.weight', 'up_blocks.1.resnets.2.temp_conv1.weight', 'down_blocks.0.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.10.norm_temp.bias', 'up_blocks.2.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.7.cross_pose_norm.weight', 'down_blocks.3.resnets.1.temp_conv2.bias', 'mid_block.attentions.0.transformer_blocks.0.norm_temp.bias', 'down_blocks.1.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.0.cross_pose_norm.bias', 'down_blocks.0.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.2.block2.bias', 'controlnet_adapter.body.4.norm_self_temp.bias', 'controlnet_adapter.body.2.attn_temp.to_v.weight', 'controlnet_adapter.body.9.norm_self_temp.weight', 'controlnet_adapter.body.5.ff.net.2.bias', 'controlnet_adapter.body.9.block2.bias', 'controlnet_adapter.body.7.ff_norm.weight', 'up_blocks.1.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.8.block1.bias', 'controlnet_adapter.body.5.block1.weight', 'controlnet_adapter.body.11.attn_self_temp.to_out.0.weight', 'up_blocks.0.resnets.2.temp_conv2.bias', 'down_blocks.1.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.8.ff.net.0.proj.bias', 'controlnet_adapter.body.11.attn_self_temp.to_v.weight', 'up_blocks.2.attentions.2.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.10.attn_self_temp.to_out.0.bias', 'up_blocks.1.attentions.2.transformer_blocks.0.attn_temp.to_v.weight', 'up_blocks.2.attentions.2.transformer_blocks.0.norm_temp.weight', 'up_blocks.3.resnets.1.temp_conv1.bias', 'controlnet_adapter.body.6.attn_pose.to_out.0.bias', 'up_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.4.block2.weight', 'controlnet_adapter.body.7.attn_temp.to_k.weight', 'controlnet_adapter.body.0.ff.net.0.proj.bias', 'controlnet_adapter.body.7.norm_self_temp.weight', 'controlnet_adapter.body.6.attn_pose.to_k.weight', 'up_blocks.3.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.8.ff.net.2.bias', 'up_blocks.2.attentions.2.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.11.norm_temp.weight', 'up_blocks.2.resnets.2.temp_conv1.bias', 'controlnet_adapter.body.7.ff.net.2.weight', 'up_blocks.0.resnets.0.temp_conv1.bias', 'down_blocks.2.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.6.attn_self_temp.to_v.weight', 'controlnet_adapter.body.5.norm_temp.weight', 'down_blocks.2.resnets.0.temp_conv2.weight', 'controlnet_adapter.body.7.norm_temp.bias', 'controlnet_adapter.body.8.attn_pose.to_v.weight', 'controlnet_adapter.body.2.block2.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.2.block1.bias', 'down_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.7.attn_pose.to_out.0.bias', 'controlnet_adapter.body.6.attn_pose.to_v.weight', 'controlnet_adapter.body.11.attn_temp.to_out.0.weight', 'down_blocks.2.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.5.norm_self_temp.weight', 'controlnet_adapter.body.4.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.5.block2.bias', 'up_blocks.1.attentions.2.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.2.attn_temp.to_out.0.bias', 'controlnet_adapter.body.9.attn_self_temp.to_out.0.bias', 'down_blocks.0.resnets.0.temp_conv1.weight', 'up_blocks.2.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.8.block2.weight', 'controlnet_adapter.body.10.attn_pose.to_out.0.weight', 'controlnet_adapter.body.5.attn_pose.to_q.weight', 'controlnet_adapter.body.0.attn_temp.to_k.weight', 'controlnet_adapter.body.0.attn_temp.to_v.weight', 'down_blocks.0.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.5.cross_pose_norm.bias', 'controlnet_adapter.body.0.block2.bias', 'controlnet_adapter.body.10.attn_self_temp.to_q.weight', 'controlnet_adapter.body.10.ff.net.2.weight', 'controlnet_adapter.body.9.norm_temp.bias', 'down_blocks.3.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.11.ff.net.0.proj.weight', 'down_blocks.0.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.2.ff.net.0.proj.bias', 'down_blocks.1.resnets.0.temp_conv2.weight', 'controlnet_adapter.body.3.norm_temp.bias', 'up_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.2.block1.weight', 'controlnet_adapter.body.3.ff.net.0.proj.weight', 'controlnet_adapter.body.1.attn_pose.to_out.0.bias', 'up_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.3.attn_temp.to_q.weight', 'controlnet_adapter.body.6.ff.net.0.proj.bias', 'controlnet_adapter.body.0.attn_pose.to_out.0.bias', 'controlnet_adapter.body.3.attn_temp.to_v.weight', 'down_blocks.0.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.5.attn_temp.to_out.0.bias', 'controlnet_adapter.body.2.ff.net.0.proj.weight', 'controlnet_adapter.body.1.attn_temp.to_out.0.bias', 'controlnet_adapter.body.4.ff.net.2.bias', 'down_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.11.attn_pose.to_q.weight', 'controlnet_adapter.body.0.norm_self_temp.bias', 'controlnet_adapter.body.0.attn_temp.to_out.0.bias', 'up_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.9.ff.net.2.bias', 'controlnet_adapter.body.9.attn_self_temp.to_v.weight', 'controlnet_adapter.body.3.attn_self_temp.to_q.weight', 'controlnet_adapter.body.0.attn_pose.to_q.weight', 'down_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.7.attn_pose.to_q.weight', 'controlnet_adapter.body.6.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.1.block2.bias', 'controlnet_adapter.body.4.attn_temp.to_k.weight', 'up_blocks.3.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'mid_block.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.1.cross_pose_norm.weight', 'controlnet_adapter.body.6.block2.bias', 'controlnet_adapter.body.3.ff_norm.weight', 'controlnet_adapter.body.7.block1.bias', 'controlnet_adapter.body.8.norm_temp.weight', 'controlnet_adapter.body.2.norm_self_temp.bias', 'up_blocks.1.resnets.2.temp_conv2.bias', 'controlnet_adapter.body.4.ff_norm.weight', 'controlnet_adapter.body.2.attn_self_temp.to_q.weight', 'mid_block.resnets.1.temp_conv1.weight', 'controlnet_adapter.body.8.cross_pose_norm.bias', 'controlnet_adapter.body.11.attn_temp.to_k.weight', 'controlnet_adapter.body.4.attn_pose.to_q.weight', 'mid_block.resnets.0.temp_conv2.bias', 'up_blocks.0.resnets.1.temp_conv2.weight', 'controlnet_adapter.body.2.attn_temp.to_out.0.weight', 'controlnet_adapter.body.4.attn_temp.to_out.0.bias', 'down_blocks.0.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.4.attn_self_temp.to_k.weight', 'controlnet_adapter.body.6.attn_pose.to_q.weight', 'controlnet_adapter.body.9.norm_self_temp.bias', 'controlnet_adapter.body.8.norm_self_temp.bias', 'down_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'up_blocks.2.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.0.ff_norm.bias', 'up_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.7.attn_temp.to_out.0.bias', 'up_blocks.3.resnets.2.temp_conv2.weight', 'controlnet_adapter.body.7.attn_temp.to_q.weight', 'controlnet_adapter.body.7.block1.weight', 'controlnet_adapter.body.4.cross_pose_norm.weight', 'down_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.7.attn_self_temp.to_q.weight', 'controlnet_adapter.body.1.attn_self_temp.to_q.weight', 'controlnet_adapter.body.11.attn_temp.to_q.weight', 'controlnet_adapter.body.2.norm_self_temp.weight', 'controlnet_adapter.body.4.attn_pose.to_out.0.weight', 'controlnet_adapter.body.10.attn_temp.to_q.weight', 'controlnet_adapter.body.2.ff_norm.bias', 'down_blocks.0.attentions.1.transformer_blocks.0.norm_temp.bias', 'down_blocks.2.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.5.attn_pose.to_k.weight', 'up_blocks.1.resnets.0.temp_conv1.bias', 'controlnet_adapter.body.10.attn_self_temp.to_v.weight', 'down_blocks.3.resnets.1.temp_conv2.weight', 'up_blocks.1.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.11.attn_temp.to_out.0.bias', 'controlnet_adapter.body.3.attn_pose.to_k.weight', 'controlnet_adapter.body.4.norm_temp.weight', 'controlnet_adapter.body.10.attn_pose.to_q.weight', 'controlnet_adapter.body.4.ff.net.0.proj.bias', 'controlnet_adapter.body.1.ff_norm.weight', 'up_blocks.2.resnets.2.temp_conv2.weight', 'controlnet_adapter.body.2.cross_pose_norm.bias', 'up_blocks.2.resnets.0.temp_conv2.bias', 'controlnet_adapter.body.9.cross_pose_norm.bias', 'controlnet_adapter.body.1.cross_pose_norm.bias', 'controlnet_adapter.body.1.norm_temp.bias', 'down_blocks.0.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'controlnet_adapter.body.5.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.2.attn_pose.to_q.weight', 'down_blocks.0.attentions.1.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.9.attn_self_temp.to_k.weight', 'controlnet_adapter.body.9.attn_pose.to_out.0.weight', 'down_blocks.0.resnets.0.temp_conv2.weight', 'controlnet_adapter.body.4.norm_self_temp.weight', 'down_blocks.0.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.5.cross_pose_norm.weight', 'down_blocks.2.resnets.0.temp_conv2.bias', 'up_blocks.3.resnets.2.temp_conv1.bias', 'controlnet_adapter.body.11.cross_pose_norm.weight', 'controlnet_adapter.body.3.block2.weight', 'up_blocks.2.attentions.2.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.7.attn_self_temp.to_v.weight', 'controlnet_adapter.body.10.attn_pose.to_out.0.bias', 'controlnet_adapter.body.3.attn_temp.to_out.0.weight', 'up_blocks.1.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.2.ff.net.2.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.attn_temp.to_out.0.bias', 'down_blocks.0.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'down_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_q.weight', 'controlnet_adapter.body.8.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.5.attn_pose.to_out.0.weight', 'controlnet_adapter.body.3.ff.net.2.weight', 'up_blocks.3.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.6.attn_pose.to_out.0.weight', 'up_blocks.1.attentions.2.transformer_blocks.0.attn_temp.to_out.0.bias', 'controlnet_adapter.body.4.cross_pose_norm.bias', 'up_blocks.3.attentions.0.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.0.attn_self_temp.to_out.0.weight', 'controlnet_adapter.body.2.attn_self_temp.to_v.weight', 'controlnet_adapter.body.5.ff.net.0.proj.weight', 'controlnet_adapter.body.9.ff_norm.weight', 'controlnet_adapter.body.6.attn_temp.to_q.weight', 'controlnet_adapter.body.11.ff_norm.bias', 'controlnet_adapter.body.9.block2.weight', 'controlnet_adapter.body.4.attn_self_temp.to_q.weight', 'controlnet_adapter.body.10.attn_self_temp.to_k.weight', 'controlnet_adapter.body.10.block1.bias', 'up_blocks.0.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.3.attn_self_temp.to_out.0.weight', 'down_blocks.1.resnets.0.temp_conv2.bias', 'up_blocks.3.attentions.1.transformer_blocks.0.attn_temp.to_out.0.weight', 'mid_block.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.8.attn_temp.to_k.weight', 'up_blocks.3.attentions.2.transformer_blocks.0.norm_temp.weight', 'down_blocks.0.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'up_blocks.2.resnets.2.temp_conv1.weight', 'controlnet_adapter.body.9.attn_pose.to_v.weight', 'controlnet_adapter.body.8.attn_pose.to_out.0.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.8.attn_self_temp.to_k.weight', 'controlnet_adapter.body.11.attn_self_temp.to_k.weight', 'down_blocks.2.attentions.0.transformer_blocks.0.attn_temp.to_v.weight', 'down_blocks.2.attentions.1.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.9.attn_temp.to_q.weight', 'controlnet_adapter.body.9.norm_temp.weight', 'controlnet_adapter.body.9.attn_pose.to_out.0.bias', 'controlnet_adapter.body.2.attn_pose.to_v.weight', 'up_blocks.0.resnets.2.temp_conv2.weight', 'down_blocks.2.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.11.norm_self_temp.weight', 'controlnet_adapter.body.7.cross_pose_norm.bias', 'controlnet_adapter.body.0.attn_temp.to_q.weight', 'controlnet_adapter.body.7.norm_temp.weight', 'controlnet_adapter.body.9.attn_pose.to_k.weight', 'controlnet_adapter.body.5.ff.net.0.proj.bias', 'up_blocks.1.resnets.1.temp_conv1.bias', 'up_blocks.2.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.6.ff_norm.weight', 'down_blocks.1.attentions.0.transformer_blocks.0.attn_temp.to_out.0.bias', 'up_blocks.0.resnets.0.temp_conv2.bias', 'controlnet_adapter.body.0.norm_self_temp.weight', 'up_blocks.3.resnets.0.temp_conv1.weight', 'controlnet_adapter.body.1.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.3.ff.net.0.proj.bias', 'controlnet_adapter.body.3.attn_pose.to_out.0.weight', 'up_blocks.1.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'controlnet_adapter.body.4.block2.bias', 'down_blocks.1.attentions.1.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.9.attn_temp.to_v.weight', 'controlnet_adapter.body.8.attn_temp.to_out.0.bias', 'controlnet_adapter.body.11.attn_pose.to_out.0.bias', 'controlnet_adapter.body.1.attn_temp.to_v.weight', 'controlnet_adapter.body.1.attn_pose.to_q.weight', 'controlnet_adapter.body.5.norm_self_temp.bias', 'controlnet_adapter.body.7.attn_pose.to_v.weight', 'controlnet_adapter.body.7.block2.bias', 'up_blocks.3.attentions.1.transformer_blocks.0.attn_temp.to_v.weight', 'down_blocks.1.resnets.1.temp_conv2.bias', 'controlnet_adapter.body.7.ff.net.0.proj.bias', 'controlnet_adapter.body.6.attn_temp.to_out.0.weight', 'controlnet_adapter.body.1.norm_self_temp.weight', 'up_blocks.3.attentions.0.transformer_blocks.0.norm_temp.bias', 'controlnet_adapter.body.4.norm_temp.bias', 'controlnet_adapter.body.1.norm_temp.weight', 'controlnet_adapter.body.3.cross_pose_norm.bias', 'controlnet_adapter.body.6.block1.bias', 'up_blocks.2.resnets.0.temp_conv1.weight', 'up_blocks.0.resnets.1.temp_conv1.weight', 'up_blocks.1.resnets.0.temp_conv2.bias', 'down_blocks.0.attentions.0.transformer_blocks.0.norm_temp.weight', 'controlnet_adapter.body.5.attn_temp.to_v.weight', 'controlnet_adapter.body.4.ff.net.0.proj.weight', 'controlnet_adapter.body.7.attn_self_temp.to_out.0.bias', 'controlnet_adapter.body.1.attn_temp.to_out.0.weight', 'up_blocks.3.attentions.1.transformer_blocks.0.attn_temp.to_k.weight', 'controlnet_adapter.body.1.block2.weight', 'controlnet_adapter.body.8.attn_self_temp.to_v.weight', 'controlnet_adapter.body.6.attn_temp.to_v.weight']

Thank you! Hoping for your earliest response! Ish

Francis-Rings commented 1 month ago

Thanks for your attention! As the Stable Diffusion is an image domain model, temporal-aware components and the additional adapter in our MotionEditor cannot inherit the original weights of SD. The weights of these additional components are randomly initialized.

DAVEISHAN commented 1 month ago

Thank you, for your reply @Francis-Rings, I am not getting results like your demos. I have sent you a detailed email regarding it. Hoping to hear from you.