search

lllyasviel / IC-Light

More relighting!
Apache License 2.0
4.24k stars 279 forks source link

Problems using Controlnet #62

Open OlegRuban-ai opened 1 month ago

OlegRuban-ai commented 1 month ago

When I try to use the original pipeline img2img - StableDiffusionImg2ImgPipeline I get the error: StableDiffusionImg2ImgPipeline.init() got an unexpected keyword argument 'controlnet'

When using the pipeline with controlnet in this way: i2i_pipe = StableDiffusionControlNetPipeline( vae=vae,

custom_pipeline="stable_diffusion_controlnet_img2img",

text_encoder=text_encoder,
tokenizer=tokenizer,
controlnet=controlnet,
unet=unet,
scheduler=dpmpp_2m_sde_karras_scheduler,
safety_checker=None,
requires_safety_checker=False,
feature_extractor=None,
image_encoder=None

).to('cuda')

I get an error when generating: RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[2, 4, 128, 128] to have 3 channels, but got 4 channels instead

After this part has been added to Unet, is it possible to add controlnet there? sd_offset = sf.load_file(model_path) sd_origin = unet.state_dict() keys = sd_origin.keys() sd_merged = {k: sd_origin[k] + sd_offset[k] for k in sd_origin.keys()} unet.load_state_dict(sd_merged, strict=True)

OlegRuban-ai commented 1 month ago

Also wondering if it is possible to use IP adapter with this approach?