import ControlNetModel_Union error

[JohnsonXi]

from diffusers.models import AutoencoderKL, ControlNetModel, ControlNetModel_Union, UNet2DConditionModel ImportError: cannot import name 'ControlNetModel_Union' from 'diffusers.models' (/root/miniconda3/envs/controlplus/lib/python3.8/site-packages/diffusers/models/init.py)

All packages follow requirements, diffusers==0.27.2. I can't find ControlNetModel_Union in google or diffusers doc.

[xinsir6]

You should git clone this repo, and use the above code to import ControlNetModel_Union, as the arichtecture is new, diffusers have not merge it.

[JohnsonXi]

My code clone from here in yesterday. This error occurs in pipeline_controlnet_union_sd_xl.py. Please check the code！ Traceback (most recent call last): File "controlnet_union_test_openpose.py", line 11, in from pipeline.pipeline_controlnet_union_sd_xl import StableDiffusionXLControlNetUnionPipeline File "/mnt/WujieAITeam/private/xtj/ControlNetPlus/pipeline/pipeline_controlnet_union_sd_xl.py", line 30, in from diffusers.models import AutoencoderKL, ControlNetModel, ControlNetModel_Union, UNet2DConditionModel ImportError: cannot import name 'ControlNetModel_Union' from 'diffusers.models' (/root/miniconda3/envs/controlplus/lib/python3.8/site-packages/diffusers/models/init.py)

[icech]

it 's a bug, you could fix the code of pipeline_controlnet_union_sd_xl.py, don't import ControlNetModel_Union from diffusers.models, but import it from models.controlnet_union

[xinsir6]

OK, I understand it. I will fix it soon.

[xinsir6]

My code clone from here in yesterday. This error occurs in pipeline_controlnet_union_sd_xl.py. Please check the code！ Traceback (most recent call last): File "controlnet_union_test_openpose.py", line 11, in from pipeline.pipeline_controlnet_union_sd_xl import StableDiffusionXLControlNetUnionPipeline File "/mnt/WujieAITeam/private/xtj/ControlNetPlus/pipeline/pipeline_controlnet_union_sd_xl.py", line 30, in from diffusers.models import AutoencoderKL, ControlNetModel, ControlNetModel_Union, UNet2DConditionModel ImportError: cannot import name 'ControlNetModel_Union' from 'diffusers.models' (/root/miniconda3/envs/controlplus/lib/python3.8/site-packages/diffusers/models/init.py)

Thank you for you reporting, I update the code, now you can try it again. Any questions just ask me.

[dahwin]

still i'm getting this error

`%cd ControlNetPlus import os import cv2 import torch import random import numpy as np from PIL import Image

from diffusers import AutoencoderKL

from diffusers import EulerAncestralDiscreteScheduler

from models.controlnet_union import ControlNetModel_Union from pipeline.pipeline_controlnet_union_sd_xl import StableDiffusionXLControlNetUnionPipeline

/kaggle/working/ControlNetPlus

ImportError Traceback (most recent call last) Cell In[1], line 10 7 from PIL import Image 8 # from diffusers import AutoencoderKL 9 # from diffusers import EulerAncestralDiscreteScheduler ---> 10 from models.controlnet_union import ControlNetModel_Union 11 from pipeline.pipeline_controlnet_union_sd_xl import StableDiffusionXLControlNetUnionPipeline

File /kaggle/working/ControlNetPlus/models/controlnet_union.py:22 19 from torch.nn import functional as F 21 from diffusers.configuration_utils import ConfigMixin, register_to_config ---> 22 from diffusers.loaders import FromOriginalControlNetMixin 23 from diffusers.utils import BaseOutput, logging 24 from diffusers.models.attention_processor import ( 25 ADDED_KV_ATTENTION_PROCESSORS, 26 CROSS_ATTENTION_PROCESSORS, (...) 29 AttnProcessor, 30 )

ImportError: cannot import name 'FromOriginalControlNetMixin' from 'diffusers.loaders' (/opt/conda/lib/python3.10/site-packages/diffusers/loaders/init.py)`

[dahwin]

[xinsir6]

It seems to be the version in your diffusers, please check it. I just checked the 0.27.2 version The FromOriginalControlNetMixin should be in the /your anaconda path/anaconda3/envs/diffusers/lib/python3.8/site-packages/diffusers/loaders/init.py file. please first check it!

[JohnsonXi]

OK, I understand it. I will fix it soon.

I try to use from ..models.controlnet_union import ControlNetModel_Union, but occurs ValueError: attempted relative import beyond top-level package error, so try to from models.controlnet_union import ControlNetModel_Union, import succeed. However, continuing to run the code will cause it to get stuck and there will be no error messages, and it will also be impossible to use the debugger. After investigation, it was found that when the program reaches the line controlnet_model = ControlNetModel_Union.from_pretrained("xinsir/controlnet-union-sdxl-1.0", torch_dtype=torch.float16, use_safetensors=True), the aforementioned issue occurs. This is running “python controlnet_union_test_anime_lineart.py” on linux terminal.

This is debugging on vscode.

[xinsir6]

OK, I understand it. I will fix it soon.

I try to use from ..models.controlnet_union import ControlNetModel_Union, but occurs ValueError: attempted relative import beyond top-level package error, so try to from models.controlnet_union import ControlNetModel_Union, import succeed. However, continuing to run the code will cause it to get stuck and there will be no error messages, and it will also be impossible to use the debugger. After investigation, it was found that when the program reaches the line controlnet_model = ControlNetModel_Union.from_pretrained("xinsir/controlnet-union-sdxl-1.0", torch_dtype=torch.float16, use_safetensors=True), the aforementioned issue occurs. This is running “python controlnet_union_test_anime_lineart.py” on linux terminal. This is debugging on vscode.

It may be the network connecting problem, long time no output, you can download the https://huggingface.co/xinsir/controlnet-union-sdxl-1.0 models first, and change the "xinsir/controlnet-union-sdxl-1.0" in your scripts to the local path you put the model, such as /home/xinsir/models/controlnet-union-sdxl-1.0, hope it can help you!

[JohnsonXi]

Checking the network by using curl to access Google and Hugging Face is not a problem, and downloading the VAE and Scheduler also goes smoothly. Could the issue be related to the from_pretrained method that is being inherited?

[xinsir6]

Have you fix it? My own offline test scripts is also used in this way, I just download the models and change the path to local path, it works well.

[JohnsonXi]

Have you fix it? My own offline test scripts is also used in this way, I just download the models and change the path to local path, it works well.

Thank you for your response. I have also successfully loaded controlnet-union-sdxl-1.0 using the local path, so the from_pretrained("xinsir/controlnet-union-sdxl-1.0") seems to have a clear bug. Currently, I don't know how to fix it. If it cannot be fixed temporarily, I hope the source code can be modified to load it locally or it can be mentioned in the readme.

[xinsir6]

Thank you for you replay, I will check it seriously today, and give a proper solution no later than this weekend.

JohnsonXi @.***> 于2024年7月11日周四 17:11写道：

Have you fix it? [image: image] https://private-user-images.githubusercontent.com/128811208/347444193-09e899ee-3566-47f7-a4c5-8c4e4d3eecc1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA2ODkyMTcsIm5iZiI6MTcyMDY4ODkxNywicGF0aCI6Ii8xMjg4MTEyMDgvMzQ3NDQ0MTkzLTA5ZTg5OWVlLTM1NjYtNDdmNy1hNGM1LThjNGU0ZDNlZWNjMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQwOTA4MzdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iMGU1ODIyYWYxOGNlMzhhYjBhMTNkMDFiZjg2NjEwMzI5ZGNkZDkzN2ZmMTlkY2Y1NWJjMGIxNmIwOWE5MmE5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.9f1-UIbF6px7FgdiFrCfwMRaJK9GHLSTVfT6fkHJmvU My own offline test scripts is also used in this way, I just download the models and change the path to local path, it works well.

Thank you for your response. I have also successfully loaded controlnet-union-sdxl-1.0 using the local path, so the from_pretrained("xinsir/controlnet-union-sdxl-1.0") seems to have a clear bug. Currently, I don't know how to fix it. If it cannot be fixed temporarily, I hope the source code can be modified to load it locally or it can be mentioned in the readme.

— Reply to this email directly, view it on GitHub https://github.com/xinsir6/ControlNetPlus/issues/7#issuecomment-2222420742, or unsubscribe https://github.com/notifications/unsubscribe-auth/A6WYBSCFN77U52EJHLWDGLTZLZD25AVCNFSM6AAAAABKR5DZUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRSGQZDANZUGI . You are receiving this because you commented.Message ID: @.***>

[xinsir6]

Have you fix it? My own offline test scripts is also used in this way, I just download the models and change the path to local path, it works well.

Thank you for your response. I have also successfully loaded controlnet-union-sdxl-1.0 using the local path, so the from_pretrained("xinsir/controlnet-union-sdxl-1.0") seems to have a clear bug. Currently, I don't know how to fix it. If it cannot be fixed temporarily, I hope the source code can be modified to load it locally or it can be mentioned in the readme.

I just re-clone my project and run it from the start, it indeed exists the import error like that: but when it run below, the xinsir/controlnet-union-sdxl-1.0 is downloaded correctly like that: I fix the relative import Error and push it to the main branch, but I still don't know why your model get stacked in downloading, so I guess this may not be a certain issue and I can't reproduct it. Thank your for you reporting! Hope you can use it well-off。

[JohnsonXi]

It's possible that others may be facing the same problem as me with the download issue, and the problem may be related to the server being used. After successfully importing the model, there are no new bugs, and I'm currently testing different cases. I found that the control effect is quite good. I previously thought that too many independent controlnets would be a hassle, and I had considered the possibility of having a more unified controlnet. I didn't expect you to implement this so quickly, and the effect is also good. This is truly an impressive piece of work. In the future, you could consider integrating with other open-source communities. Thank you for your effort! Best wishes!

[xinsir6]

Thank you for your approval! I will do my best to achieve good models for opensource community! Perhaps someday I will open a company, hope those of you can come and support, it would be my pleasure to let you enjoy it.

[JohnsonXi]

I can't wait to see your products, and I hope to see your name on the AIGC elite ranking.

Thank you for your approval! I will do my best to achieve good models for opensource community! Perhaps someday I will open a company, hope those of you can come and support, it would be my pleasure to let you enjoy it.

[bonlime]

if anyone faces the problem with FromOriginalControlNetMixin you can replace it with FromOriginalModelMixin everywhere in this code base, and then reaplce from diffusers.models.unet_2d_blocks import with from diffusers.models.unets.unet_2d_blocks import and everything would work on the latest diffusers version