search

vislearn / ControlNet-XS

Apache License 2.0
411 stars 12 forks source link

Weird results after updating controlnet-xs + sdxl from controlnet + sdxl #8

Closed LoveU3tHousand2 closed 9 months ago

LoveU3tHousand2 commented 9 months ago

I have been fine-tuning sdxl with controlnet for remote-sensing images of buildings and the seg-map as control, but when I tried your work , the sample results became weird... This is my controlnet+sdxl result below: image And this is controlnet-XS + sdxl: image We can see difference between those two results, controlnet is more natural and controlnet-xs is weird. Is that normal or I missed some detail ?

Sipirius commented 9 months ago

Because of the vast redundancy in parameters in the original ControlNet, it can be a little easier to finetune to different tasks. From the looks of it you would "just" have to find the right training parameters.

Cheers

LoveU3tHousand2 commented 9 months ago

Because of the vast redundancy in parameters in the original ControlNet, it can be a little easier to finetune to different tasks. From the looks of it you would "just" have to find the right training parameters.

Cheers

Thanks for explain, another question I want to know is how to initialize the weight of controlnex-xs if we don't copy it from unet? all zero or ... ?

Sipirius commented 9 months ago

The initialisation schedule is the same as for all other modules of the StableDiffusion U-Net. So some weights are initialised randomly, some with zeros.

Cheers