lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
30.67k stars 2.75k forks source link

Question about `sd_locked` in ControlNet 1.0 and 1.1 #625

Open xyfJASON opened 11 months ago

xyfJASON commented 11 months ago

In the paper, it is said that the parameters of Stable Diffusion(SD) are locked when training ControlNet. However, when I looked into the released ControlNet 1.0 checkpoints, I found that the weights of SD UNet decoder are different from SD1.5 (v1-5-pruned.ckpt). I notice that there's an option sd_locked that enables training SD UNet decoder. So does this mean that ControlNet 1.0 are trained with sd_locked=False?

Furthermore, different from ControlNet 1.0 checkpoints, the ControlNet 1.1 checkpoints don't contain SD weights and need to be used along with v1-5-pruned.ckpt, does this mean that ControlNet 1.1 are trained with sd_locked=True?

Here is the script I used to compare two checkpoints:

import torch

ckpt = torch.load('./ckpts/v1-5-pruned.ckpt', map_location='cpu')['state_dict']
ckpt2 = torch.load('./ckpts/control_sd15_canny.pth', map_location='cpu')

notclose = []
maxdiff = 0.
for k, v in ckpt.items():
    if k.startswith('model') and k in ckpt2:
        if not torch.allclose(ckpt2[k], ckpt[k]):
            notclose.append(k)
            maxdiff = max(maxdiff, torch.max(torch.abs(ckpt2[k] - ckpt[k])).item())

print('notclose:')
print('\n'.join(notclose))
print(f'maxdiff: {maxdiff}')