search

TheDenk / cogvideox-controlnet

Simple Controlnet module for CogvideoX model.
Apache License 2.0
62 stars 2 forks source link

load controlnet model error #9

Open MianWang123 opened 1 day ago

MianWang123 commented 1 day ago

Hi, I was trying to finetune controlnet based on your checkpoint, but there's always a mismatch between the current model and your checkpoint. The checkpoint I used was "cogvideox-5b-controlnet-canny-v1", base model was "CogVideoX-5b", do you know where the problem could be?

![Uploading error.jpg…]()

MianWang123 commented 1 day ago

IMG_3194

MianWang123 commented 13 hours ago

Update!! For cogvideox-5b-controlnet-canny-v1 model, transformer_num_layers=18 and out_proj_dim=3072, which differs from cogvideox-2b-controlnet-canny-v1 model. But the base model has to be CogVideoX-2b for either of the above 2 checkpoints.

TheDenk commented 9 hours ago

Hey :) 5b and 2b models have the same base model (2b), but 5b transformer have different inner dimention and because of this controlnet for 5b was created with additional output layer which is mapping 2b dim to 5b.