search

vladmandic / automatic

SD.Next: Advanced Implementation of Stable Diffusion and other Diffusion-based generative image models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.48k stars 400 forks source link

[Issue]: Controlnet: mat1 and mat2 must have the same dtype #2662

Closed gruevy closed 8 months ago

gruevy commented 8 months ago

Issue Description

Happens regardless of SD model, controlnet model, or sampler.

Any time I use controlnet, after inputting my reference image, selecting my models, putting in the prompt, and hitting generate, it fails with the error

"mat1 and mat2 must have the same dtype"

I searched this github and found this, which is the only mention of "mat1" that appears. It may be useful. https://github.com/vladmandic/automatic/issues/515

I don't have a fresh install but I'll try one later and amend this issue if I find anything useful. I haven't tried adjusting any other settings after this happened the first time.

Version Platform Description

app: SD.next updated: 2023-12-29 hash: f4d4f8da url: https://github.com/vladmandic/automatic/tree/master

Running on windows 11 with a 4090

Relevant log output

PS D:\stablediffusion\automatic> .\webui.bat --listen --debug
Using VENV: D:\stablediffusion\automatic\venv
12:16:16-401212 INFO     Starting SD.Next
12:16:16-404212 INFO     Logger: file="D:\stablediffusion\automatic\sdnext.log" level=DEBUG size=65 mode=create
12:16:16-405212 INFO     Python 3.10.7 on Windows
12:16:16-549767 INFO     Version: app=sd.next updated=2023-12-29 hash=f4d4f8da
                         url=https://github.com/vladmandic/automatic/tree/master
12:16:16-938060 INFO     Latest published version: c77505fa6f35fd4540abfbe9b2d326b299203461 2023-12-29T17:14:09Z
12:16:16-945060 INFO     Platform: arch=AMD64 cpu=AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD system=Windows
                         release=Windows-10-10.0.22621-SP0 python=3.10.7
12:16:16-947061 DEBUG    Setting environment tuning
12:16:16-948061 DEBUG    Cache folder: C:\Users\Ryan\.cache\huggingface\hub
12:16:16-949061 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
12:16:16-950062 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
12:16:16-952062 INFO     nVidia CUDA toolkit detected: nvidia-smi present
12:16:17-012076 WARNING  Modified files: ['repositories/BLIP/BLIP.gif', 'repositories/CodeFormer/.gitignore',
                         'scripts/detect_extension.py']
12:16:17-053094 DEBUG    Repository update time: Fri Dec 29 07:32:52 2023
12:16:17-054094 INFO     Startup: standard
12:16:17-055094 INFO     Verifying requirements
12:16:17-068098 INFO     Verifying packages
12:16:17-070098 INFO     Verifying submodules
12:16:19-558966 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main
12:16:19-643487 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main
12:16:19-719504 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main
12:16:19-819029 DEBUG    Submodule: extensions-builtin/sd-webui-controlnet / main
12:16:19-935061 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-images-browser / main
12:16:20-010077 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
12:16:20-088599 DEBUG    Submodule: modules/k-diffusion / master
12:16:20-161119 DEBUG    Submodule: modules/lora / main
12:16:20-400176 DEBUG    Submodule: wiki / master
12:16:20-450194 DEBUG    Register paths
12:16:20-586236 DEBUG    Installed packages: 234
12:16:20-588236 DEBUG    Extensions all: ['clip-interrogator-ext', 'LDSR', 'Lora', 'ScuNET', 'sd-dynamic-thresholding',
                         'sd-extension-chainner', 'sd-extension-system-info', 'sd-webui-agent-scheduler',
                         'stable-diffusion-webui-images-browser', 'stable-diffusion-webui-rembg', 'SwinIR']
12:16:20-590237 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions-builtin\clip-interrogator-ext\install.py
12:16:24-292709 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions-builtin\sd-extension-system-info\install.py
12:16:24-752336 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions-builtin\sd-webui-agent-scheduler\install.py
12:16:25-218514 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions-builtin\stable-diffusion-webui-images-browser\install.p
                         y
12:16:25-662424 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions-builtin\stable-diffusion-webui-rembg\install.py
12:16:26-296596 DEBUG    Extensions all: ['adetailer', 'canvas-zoom', 'sd-webui-reactor']
12:16:26-298598 DEBUG    Running extension installer: D:\stablediffusion\automatic\extensions\adetailer\install.py
12:16:26-820055 DEBUG    Running extension installer: D:\stablediffusion\automatic\extensions\canvas-zoom\install.py
12:16:29-134434 DEBUG    Running extension installer:
                         D:\stablediffusion\automatic\extensions\sd-webui-reactor\install.py
12:16:29-675448 INFO     Extensions enabled: ['clip-interrogator-ext', 'LDSR', 'Lora', 'ScuNET',
                         'sd-dynamic-thresholding', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg', 'SwinIR', 'adetailer', 'canvas-zoom', 'sd-webui-reactor']
12:16:29-677449 INFO     Verifying requirements
12:16:29-693452 DEBUG    Setup complete without errors: 1703877390
12:16:29-699454 INFO     Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
12:16:29-701454 DEBUG    Starting module: <module 'webui' from 'D:\\stablediffusion\\automatic\\webui.py'>
12:16:29-702454 INFO     Command line args: ['--listen', '--debug'] listen=True debug=True
12:16:29-704455 DEBUG    Env flags: []
12:16:33-587860 INFO     Load packages: torch=2.0.1+cu118 diffusers=0.25.0 gradio=3.43.2
12:16:34-195590 DEBUG    Read: file="config.json" json=58 bytes=3176 time=0.000
12:16:34-197591 DEBUG    Unknown settings: ['hidden_tabs', 'realesrgan_enabled_models', 'multiple_tqdm',
                         'chainner_models_path']
12:16:34-199591 INFO     Engine: backend=Backend.DIFFUSERS compute=cuda mode=no_grad device=cuda
                         cross-optimization="Scaled-Dot-Product"
12:16:34-252108 INFO     Device: device=NVIDIA GeForce RTX 4090 n=1 arch=compute_37 cap=(8, 9) cuda=11.8 cudnn=8700
                         driver=546.33
12:16:37-027174 DEBUG    Entering start sequence
12:16:37-031175 DEBUG    Initializing
12:16:37-033175 INFO     Available VAEs: path="D:\stablediffusion\models\VAE" items=1
12:16:37-035175 INFO     Disabled extensions: ['a1111-sd-webui-lycoris', 'multidiffusion-upscaler-for-automatic1111',
                         'sd-webui-controlnet']
12:16:37-037679 DEBUG    Scanning diffusers cache: models\Diffusers D:\stablediffusion\models\Diffusers items=4
                         time=0.00
12:16:37-038680 DEBUG    Read: file="cache.json" json=2 bytes=11711 time=0.000
12:16:37-044680 DEBUG    Read: file="metadata.json" json=114 bytes=404392 time=0.004
12:16:37-047681 INFO     Available models: path="D:\stablediffusion\models\Stable-diffusion" items=20 time=0.01
12:16:37-321778 DEBUG    Load extensions
12:16:37-634868 INFO     Extension: script='scripts\faceid.py' [2;36m12:16:37-633869[0m[2;36m [0m[1;31mERROR   [0m
                         FaceID: No module named [32m'ip_adapter'[0m
12:16:38-815262 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         ======
12:16:38-816262 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         Error! UniPC sampler support failed to load! Is your WebUI up to date?
12:16:38-818263 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         (Error: module 'modules.models.diffusion.uni_pc' has no attribute 'sampler')
12:16:38-819263 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         ======
12:16:38-820264 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         ======
12:16:38-821265 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         Warning! Using legacy KDiff version! Is your WebUI up to date?
12:16:38-822264 INFO     Extension: script='extensions-builtin\sd-dynamic-thresholding\scripts\dynamic_thresholding.py'
                         ======
12:16:40-713131 INFO     Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer initialized.
                         version: 23.9.3, num models: 9
12:16:40-733136 STATUS   Extensions time: 3.41 { automatic=0.33 clip-interrogator-ext=1.11 sd-extension-chainner=0.14
                         sd-webui-agent-scheduler=0.33 stable-diffusion-webui-images-browser=0.20
                         stable-diffusion-webui-rembg=0.57 adetailer=0.64 }
12:16:40-773649 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2672 time=0.000
12:16:40-774649 STATUS   Upscaler create: folder="D:\stablediffusion\models\chaiNNer"
12:16:40-776649 DEBUG    Upscaler type=ESRGAN folder="D:\stablediffusion\models\ESRGAN" model="4x-UltraSharp"
                         path="D:\stablediffusion\models\ESRGAN\4x-UltraSharp.pth"
12:16:40-777649 DEBUG    Upscaler type=ESRGAN folder="D:\stablediffusion\models\ESRGAN" model="ESRGAN_4x"
                         path="D:\stablediffusion\models\ESRGAN\ESRGAN_4x.pth"
12:16:40-778650 DEBUG    Upscaler type=ESRGAN folder="D:\stablediffusion\models\ESRGAN"
                         model="x1_ITF_SkinDiffDetail_Lite_v1"
                         path="D:\stablediffusion\models\ESRGAN\x1_ITF_SkinDiffDetail_Lite_v1.pth"
12:16:40-780650 DEBUG    Upscaler type=SCUNet folder="D:\stablediffusion\models\ScuNET" model="ScuNET"
                         path="D:\stablediffusion\models\ScuNET\ScuNET.pth"
12:16:40-781651 DEBUG    Upscaler type=SwinIR folder="D:\stablediffusion\models\SwinIR" model="SwinIR_4x"
                         path="D:\stablediffusion\models\SwinIR\SwinIR_4x.pth"
12:16:40-783651 DEBUG    Load upscalers: total=48 downloaded=12 user=5 time=0.05 ['None', 'Lanczos', 'Nearest',
                         'ChaiNNer', 'ESRGAN', 'LDSR', 'RealESRGAN', 'SCUNet', 'SD', 'SwinIR']
12:16:40-797654 DEBUG    Load styles: folder="D:\stablediffusion\models\styles" items=289 time=0.01
12:16:40-801656 DEBUG    Creating UI
12:16:40-803656 STATUS   Load UI theme: name="black-teal" style=Dark base=sdnext.css
12:16:40-823663 DEBUG    Read: file="html\reference.json" json=31 bytes=16496 time=0.001
12:16:41-101233 DEBUG    Extra networks: page='model' items=51 subfolders=3 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\Stable-diffusion',
                         'D:\\stablediffusion\\models\\Diffusers', 'models\\Reference'] list=0.01 desc=0.00 info=0.00
                         workers=2
12:16:41-116237 DEBUG    Extra networks: page='style' items=289 subfolders=2 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\styles', 'html'] list=0.01 desc=0.00 info=0.00 workers=2
12:16:41-119237 DEBUG    Extra networks: page='embedding' items=5 subfolders=1 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\embeddings'] list=0.00 desc=0.00 info=0.00 workers=2
12:16:41-121238 DEBUG    Extra networks: page='hypernetwork' items=0 subfolders=1 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\hypernetworks'] list=0.00 desc=0.00 info=0.00 workers=2
12:16:41-123238 DEBUG    Extra networks: page='vae' items=1 subfolders=1 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\VAE'] list=0.00 desc=0.00 info=0.00 workers=2
12:16:41-126239 DEBUG    Extra networks: page='lora' items=8 subfolders=1 tab=txt2img
                         folders=['D:\\stablediffusion\\models\\Lora', 'D:\\stablediffusion\\models\\LyCORIS'] list=0.25
                         desc=0.00 info=0.00 workers=2
12:16:41-325212 DEBUG    Control initialize: models=models\control
12:16:41-528268 DEBUG    Read: file="ui-config.json" json=6 bytes=213 time=0.000
12:16:41-649303 DEBUG    Themes: builtin=9 default=5 external=51
12:16:42-098423 DEBUG    Script: 0.36 ui_tabs
                         D:\stablediffusion\automatic\extensions-builtin\stable-diffusion-webui-images-browser\scripts\i
                         mage_browser.py
12:16:42-102424 DEBUG    Read: file="D:\stablediffusion\automatic\html\extensions.json" json=243 bytes=146791 time=0.001
12:16:43-766623 DEBUG    Extension list: processed=234 installed=17 enabled=14 disabled=3 visible=234 hidden=0
12:16:48-171177 STATUS   Local URL: http://localhost:7860/
12:16:48-172178 DEBUG    Gradio functions: registered=2076
12:16:48-173178 STATUS   Initializing middleware
12:16:48-177178 DEBUG    Creating API
12:16:48-326719 STATUS   [AgentScheduler] Task queue is empty
12:16:48-327719 STATUS   [AgentScheduler] Registering APIs
12:16:48-445753 DEBUG    Scripts setup: ['X/Y/Z Grid:0.007', 'ADetailer:0.025', 'ReActor:0.005']
12:16:48-446753 DEBUG    Model metadata: file="metadata.json" no changes
12:16:48-447753 DEBUG    Model auto load disabled
12:16:48-449754 DEBUG    Save: file="config.json" json=58 bytes=3083 time=0.001
12:16:48-450754 DEBUG    Unused settings: ['hidden_tabs', 'realesrgan_enabled_models', 'multiple_tqdm']
12:16:48-451754 STATUS   Startup time: 18.74 { torch=2.91 gradio=0.93 libraries=3.44 extensions=3.41 face-restore=0.27
                         upscalers=0.05 ui-extra-networks=0.41 ui-txt2img=0.06 ui-img2img=0.07 ui-settings=0.21
                         ui-extensions=2.07 ui-defaults=0.08 launch=4.32 api=0.08 app-started=0.19 }
12:17:23-297517 STATUS   MOTD: N/A
12:17:26-620510 DEBUG    Themes: builtin=9 default=5 external=51
12:17:28-427529 STATUS   Browser session: user=None client=166.70.38.195 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64)
                         AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
12:17:32-987087 STATUS   Select: model="sd_xl_base_1.0 [31e35c80fc]"
12:17:32-989087 DEBUG    Load model weights: existing=False
                         target=D:\stablediffusion\models\Stable-diffusion\sd_xl_base_1.0.safetensors info=None
Loading model: D:\stablediffusion\models\Stable-diffusion\sd_xl_base_1.0.safetensors ━━━━━━━━━━━━━━━━ 0.0/6.9 GB -:--:--
12:17:33-724777 DEBUG    Desired Torch parameters: dtype=FP32 no-half=False no-half-vae=False upscast=False
12:17:33-725777 STATUS   Setting Torch parameters: device=cuda dtype=torch.float32 vae=torch.float32 unet=torch.float32
                         context=no_grad fp16=False bf16=False
12:17:33-726777 DEBUG    Diffusers loading: path="D:\stablediffusion\models\Stable-diffusion\sd_xl_base_1.0.safetensors"
12:17:33-727777 STATUS   Autodetect: model="Stable Diffusion XL" class=StableDiffusionXLPipeline
                         file="D:\stablediffusion\models\Stable-diffusion\sd_xl_base_1.0.safetensors" size=6617MB
12:18:00-235934 DEBUG    Server: alive=True jobs=0 requests=43 uptime=86 memory=17.73/63.92 backend=Backend.DIFFUSERS
                         state=idle
12:18:09-869507 DEBUG    Setting model: pipeline=StableDiffusionXLPipeline config={'low_cpu_mem_usage': True,
                         'torch_dtype': torch.float32, 'load_connected_pipeline': True, 'extract_ema': True,
                         'force_zeros_for_empty_prompt ': True, 'requires_aesthetics_score': False, 'use_safetensors':
                         True}
12:18:09-871508 DEBUG    Setting model VAE: name=None upcast=True
12:18:09-872509 DEBUG    Setting model: enable VAE slicing
12:18:09-873509 DEBUG    Setting model: enable VAE tiling
12:18:13-992421 STATUS   Load embeddings: loaded=5 skipped=0 time=0.10
12:18:14-627582 DEBUG    gc: collected=772 device=cuda {'ram': {'used': 1.16, 'total': 63.92}, 'gpu': {'used': 14.86,
                         'total': 23.99}, 'retries': 0, 'oom': 0}
12:18:14-637584 STATUS   Load model: time=41.00 { load=41.00 } native=1024 {'ram': {'used': 1.16, 'total': 63.92},
                         'gpu': {'used': 14.86, 'total': 23.99}, 'retries': 0, 'oom': 0}
12:18:14-644586 DEBUG    Save: file="config.json" json=58 bytes=3081 time=0.000
12:18:14-646586 DEBUG    Unused settings: ['hidden_tabs', 'realesrgan_enabled_models', 'multiple_tqdm']
12:18:14-647587 DEBUG    Setting changed: key=sd_model_checkpoint, value=sd_xl_base_1.0 [31e35c80fc]
12:18:32-997389 DEBUG    Control input: type=PIL.Image input=[<PIL.Image.Image image mode=RGB size=895x1322 at
                         0x247833A6B30>]
12:18:47-828585 DEBUG    Control processor loading: id="Canny" class=CannyDetector
12:18:47-830585 DEBUG    Control processor loaded: id="Canny" class=CannyDetector time=0.00
12:18:53-198467 DEBUG    Control ControlNet model loading: id="Canny XL" path="diffusers/controlnet-canny-sdxl-1.0"
12:18:56-082332 DEBUG    Control ControlNet model loaded: id="Canny XL" path="diffusers/controlnet-canny-sdxl-1.0"
                         time=2.88
12:19:03-512061 DEBUG    Control: tab=controlnet job=task(pahiq2ve6i6zi9j) args=(0, 'a young girl sitting on a rock',
                         '', [], 20, 0, -1.0, -1.0, 0, 0, 0, 4, 1, 4, 0.7, True, False, False, False, 4, 0.95, False, 1,
                         1, False, 0.6, 1, 1, 'None', 512, 512, 1, 0, 'Before', 0.5, 1, 1, 0, 'None', 2, True, 1, 0,
                         'none', 0.5, None, 0)
12:19:03-520063 DEBUG    Control ControlNet unit: process=Canny model=Canny XL strength=1.0 guess=False start=0 end=1
12:19:03-562073 DEBUG    Control ControlNet pipeline: class=StableDiffusionXLControlNetPipeline time=0.04
12:19:03-593585 DEBUG    Setting model VAE: name=None upcast=True
12:19:03-595586 DEBUG    Setting model: enable VAE slicing
12:19:03-596587 DEBUG    Setting model: enable VAE tiling
12:19:03-654599 DEBUG    Control processor: id="Canny" args={'low_threshold': 100, 'high_threshold': 200} time=0.03
12:19:03-706114 DEBUG    Sampler default EulerDiscreteScheduler: {'num_train_timesteps': 1000, 'beta_start': 0.00085,
                         'beta_end': 0.012, 'beta_schedule': 'scaled_linear', 'trained_betas': None, 'prediction_type':
                         'epsilon', 'interpolation_type': 'linear', 'use_karras_sigmas': False, 'sigma_min': None,
                         'sigma_max': None, 'timestep_spacing': 'leading', 'timestep_type': 'discrete', 'steps_offset':
                         1, 'rescale_betas_zero_snr': False, 'sample_max_value': 1.0, 'set_alpha_to_one': False,
                         'skip_prk_steps': True, 'clip_sample': False}
12:19:04-227420 DEBUG    Diffuser pipeline: StableDiffusionXLControlNetPipeline task=DiffusersTaskType.TEXT_2_IMAGE
                         set={'prompt_embeds': torch.Size([1, 77, 2048]), 'pooled_prompt_embeds': torch.Size([1, 1280]),
                         'negative_prompt_embeds': torch.Size([1, 77, 2048]), 'negative_pooled_prompt_embeds':
                         torch.Size([1, 1280]), 'guidance_scale': 4, 'generator': device(type='cuda'), 'output_type':
                         'latent', 'num_inference_steps': 20, 'eta': 1.0, 'width': 512, 'height': 512,
                         'controlnet_conditioning_scale': 1.0, 'control_guidance_start': 0.0, 'control_guidance_end':
                         1.0, 'guess_mode': False, 'image': <class 'PIL.Image.Image'>, 'parser': 'Full parser'}
Progress ?it/s                                              0% 0/20 00:00 ? Base
12:19:04-789504 ERROR    Processing: args={'prompt_embeds': tensor([[[-3.8923, -2.5116,  4.7176,  ...,  0.1906,  0.4180,
                         -0.2969],
                                  [ 0.0894, -0.6197, -0.4879,  ...,  0.5010, -0.0373, -0.1578],
                                  [ 0.1395, -0.0738, -0.7839,  ...,  0.5601,  0.5450,  0.2129],
                                  ...,
                                  [-0.2347,  0.7833, -0.8024,  ...,  0.4085,  0.3839,  0.3598],
                                  [-0.2480,  0.7667, -0.7941,  ...,  0.3550,  0.2441,  0.2931],
                                  [-0.2149,  0.8224, -0.7352,  ...,  0.0789,  0.3787,  0.2783]]],
                                device='cuda:0'), 'pooled_prompt_embeds': tensor([[-0.1022,  1.0178,  0.4479,  ...,
                         -1.2839, -1.0997,  1.2693]],
                                device='cuda:0'), 'negative_prompt_embeds': tensor([[[-3.8923, -2.5116,  4.7176,  ...,
                         0.1906,  0.4180, -0.2969],
                                  [-0.3756, -0.6835, -0.4727,  ...,  0.3298, -0.5163,  0.6990],
                                  [-0.4841, -0.7412, -0.4380,  ..., -0.4167,  0.3293, -0.0310],
                                  ...,
                                  [-0.0096, -0.3304, -0.3503,  ...,  0.3247,  0.0552,  0.3204],
                                  [-0.0135, -0.3250, -0.3414,  ...,  0.3370, -0.0781,  0.2833],
                                  [-0.0233, -0.2356, -0.2247,  ...,  0.3663, -0.1265,  0.3831]]],
                                device='cuda:0'), 'negative_pooled_prompt_embeds': tensor([[-0.2935,  0.8793, -0.9181,
                         ..., -0.6774, -0.3751,  0.6520]],
                                device='cuda:0'), 'guidance_scale': 4, 'generator': [<torch._C.Generator object at
                         0x00000247842AA070>], 'output_type': 'latent', 'callback_on_step_end': <function
                         process_diffusers.<locals>.diffusers_callback at 0x0000024784B5C790>,
                         'callback_on_step_end_tensor_inputs': ['latents', 'prompt_embeds', 'negative_prompt_embeds'],
                         'num_inference_steps': 20, 'eta': 1.0, 'width': 512, 'height': 512,
                         'controlnet_conditioning_scale': 1.0, 'control_guidance_start': 0.0, 'control_guidance_end':
                         1.0, 'guess_mode': False, 'image': <PIL.Image.Image image mode=L size=895x1322 at
                         0x247833A6920>} mat1 and mat2 must have the same dtype
12:19:04-797506 ERROR    Processing: RuntimeError
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ D:\stablediffusion\automatic\modules\processing_diffusers.py:450 in process_diffusers                                │
│                                                                                                                      │
│   449 │   │   t0 = time.time()                                                                                       │
│ ❱ 450 │   │   output = shared.sd_model(**base_args) # pylint: disable=not-callable                                   │
│   451 │   │   downcast_openvino(op="base")                                                                           │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\torch\utils\_contextlib.py:115 in decorate_context               │
│                                                                                                                      │
│   114 │   │   with ctx_factory():                                                                                    │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                                       │
│   116                                                                                                                │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet_sd_xl.py:1348 │
│ in __call__                                                                                                          │
│                                                                                                                      │
│   1347 │   │   │   │                                                                                                 │
│ ❱ 1348 │   │   │   │   down_block_res_samples, mid_block_res_sample = self.controlnet(                               │
│   1349 │   │   │   │   │   control_model_input,                                                                      │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl                    │
│                                                                                                                      │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                       │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                                              │
│   1502 │   │   # Do not call functions when jit is used                                                              │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\diffusers\models\controlnet.py:753 in forward                    │
│                                                                                                                      │
│   752 │   │                                                                                                          │
│ ❱ 753 │   │   emb = self.time_embedding(t_emb, timestep_cond)                                                        │
│   754 │   │   aug_emb = None                                                                                         │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl                    │
│                                                                                                                      │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                       │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                                              │
│   1502 │   │   # Do not call functions when jit is used                                                              │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\diffusers\models\embeddings.py:228 in forward                    │
│                                                                                                                      │
│   227 │   │   │   sample = sample + self.cond_proj(condition)                                                        │
│ ❱ 228 │   │   sample = self.linear_1(sample)                                                                         │
│   229                                                                                                                │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl                    │
│                                                                                                                      │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                       │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                                              │
│   1502 │   │   # Do not call functions when jit is used                                                              │
│                                                                                                                      │
│ D:\stablediffusion\automatic\venv\lib\site-packages\torch\nn\modules\linear.py:114 in forward                        │
│                                                                                                                      │
│   113 │   def forward(self, input: Tensor) -> Tensor:                                                                │
│ ❱ 114 │   │   return F.linear(input, self.weight, self.bias)                                                         │
│   115                                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: mat1 and mat2 must have the same dtype
12:19:05-134856 STATUS   Processed: images=0 time=1.43 its=0.00 memory={'ram': {'used': 1.41, 'total': 63.92}, 'gpu':
                         {'used': 20.44, 'total': 23.99}, 'retries': 0, 'oom': 0}
12:19:05-161861 STATUS   Control: pipeline units=1 process=1 time=1.64 init=0.04 proc=0.13 ctrl=1.47 outputs=0

Backend

Diffusers

Branch

Master

Model

SD-XL

Acknowledgements

vladmandic commented 8 months ago

i'll add a force cast to correct dtype when loading controlnet module. but why are you running with fp32 on rtx4090 ?!

gruevy commented 8 months ago

Thanks for the fix! The reason I switch to fp32 is that the opendalle models threw errors. I changed it and the errors went away. But I just changed it back to fp16 and the errors didn't resume, so it might have just been some odd quirk, or maybe you fixed that too.