[FluxMultiControlNetModel] object has no attribute 'config

darhsu commented 1 month ago

Describe the bug

This commit on main https://github.com/huggingface/diffusers/commit/14a1b86fc7de53ff1dbf803f616cbb16ad530e45 seems to have broke FluxMultiControlNetModel. Reverting this commit fixes the issue on line pipeline_flux_controlnet.py:844

Referencing this issue: https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro/discussions/20

Reproduction

base_model = "/archive/me_flux_shallow/FLUX.1-dev"
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro'

controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
controlnet = FluxMultiControlNetModel([controlnet_union])

pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda:0")

prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'
control_image_depth = load_image("/home/me/repos/flux_control_net/assets_depth.jpg")
control_mode_depth = 2

control_image_canny = load_image("/home/me/repos/flux_control_net/assets_canny.jpg")
control_mode_canny = 0

width, height = control_image_depth.size

image = pipe(
    prompt, 
    control_image=[control_image_depth, control_image_canny],
    control_mode=[control_mode_depth, control_mode_canny],
    width=width,
    height=height,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=24, 
    guidance_scale=3.5,
    generator=torch.manual_seed(42),
).images[0]

Logs

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 1
----> 1 image = pipe(
      2     prompt, 
      3     control_image=[control_image_depth, control_image_canny],
      5     control_mode=[control_mode_depth, control_mode_canny],
      6     width=width,
      7     height=height,
      8     controlnet_conditioning_scale=0.6,
      9     num_inference_steps=24, 
     10     guidance_scale=3.5,
     11     generator=torch.manual_seed(42),
     12 ).images[0]

File ~/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File ~/.local/lib/python3.10/site-packages/diffusers/pipelines/flux/pipeline_flux_controlnet.py:844, in FluxControlNetPipeline.__call__(self, prompt, prompt_2, height, width, num_inference_steps, timesteps, guidance_scale, control_image, control_mode, controlnet_conditioning_scale, num_images_per_prompt, generator, latents, prompt_embeds, pooled_prompt_embeds, output_type, return_dict, joint_attention_kwargs, callback_on_step_end, callback_on_step_end_tensor_inputs, max_sequence_length)
    840 # broadcast to batch dimension in a way that's compatible with ONNX/Core ML
    841 timestep = t.expand(latents.shape[0]).to(latents.dtype)
    843 guidance = (
--> 844     torch.tensor([guidance_scale], device=device) if self.controlnet.config.guidance_embeds else None
    845 )
    846 guidance = guidance.expand(latents.shape[0]) if guidance is not None else None
    848 # controlnet

File ~/.local/lib/python3.10/site-packages/diffusers/models/modeling_utils.py:151, in ModelMixin.__getattr__(self, name)
    148     return self._internal_dict[name]
    150 # call PyTorch's https://pytorch.org/docs/stable/_modules/torch/nn/modules/module.html#Module
--> 151 return super().__getattr__(name)

File ~/.local/lib/python3.10/site-packages/torch/nn/modules/module.py:1729, in Module.__getattr__(self, name)
   1727     if name in modules:
   1728         return modules[name]
-> 1729 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'FluxMultiControlNetModel' object has no attribute 'config'

System Info

🤗 Diffusers version: 0.31.0.dev0
Platform: Linux-5.10.225-213.878.amzn2.x86_64-x86_64-with-glibc2.31
Running on Google Colab?: No
Python version: 3.11.9
PyTorch version (GPU?): 2.4.1+cu121 (True)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Huggingface_hub version: 0.25.0
Transformers version: 4.44.2
Accelerate version: 0.34.2
PEFT version: 0.12.0
Bitsandbytes version: not installed
Safetensors version: 0.4.5
xFormers version: 0.0.28.post1
Accelerator: NVIDIA L40S, 46068 MiB
Using GPU in script?:
Using distributed or parallel set-up in script?:

Who can help?

@sayakpaul @yiyixuxu @DN6

also @liamdonnellymv for the original issue on huggingface

yiyixuxu commented 1 month ago

hey sorry we will merge this one soon https://github.com/huggingface/diffusers/pull/9507 that will fix this

burgalon commented 3 weeks ago

@yiyixuxu any I seem to be encountering this with FluxControlNetInpaintPipeline with latest main diffusers

yiyixuxu commented 3 weeks ago

yes! would you be interested in helping us fix it for inpaint? would need similar change to this I think https://github.com/huggingface/diffusers/pull/9586/files

nemoooooooooo commented 4 days ago

@darhsu try this: base_model = 'black-forest-labs/FLUX.1-dev' controlnet_model_union = 'InstantX/FLUX.1-dev-Controlnet-Union'

controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16) controlnet = FluxMultiControlNetModel([controlnet_union]) # we always recommend loading via FluxMultiControlNetModel

controlnet.config = controlnet_union.config

pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)

huggingface / diffusers