mat1 and mat2 must have the same dtype

[QoungYoung]

Hello! I've been using your node, but I've encountered some issues: The model I'm using is: v2-1_768-ema-pruned.ckpt The stableSR model I'm using is: webui_786v_139.ckpt The images I've tested are in png format.

The issues I've encountered are as follows:

Error occurred when executing StableSRUpscalerPipe:

mat1 and mat2 must have the same dtype

File "D:\gitProject\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "D:\gitProject\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "D:\gitProject\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "D:\gitProject\ComfyUI\custom_nodes\Comfyui-StableSR-main\nodes.py", line 72, in doit_pipe upscale_image, color_map_image = upscaler.sample(image) File "D:\gitProject\ComfyUI\custom_nodes\Comfyui-StableSR-main\modules\stablesr.py", line 268, in sample nodes.common_ksampler(sd_model, seed, steps, cfg, sampler_name, scheduler, positive, negative, empty_latent, denoise)[0] File "D:\gitProject\ComfyUI\nodes.py", line 1206, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "D:\gitProject\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 22, in informative_sample raise e File "D:\gitProject\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\hacky.py", line 9, in informative_sample return original_sample(args, kwargs) File "D:\gitProject\ComfyUI\comfy\sample.py", line 97, in sample samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 785, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 690, in sample samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 630, in sample samples = getattr(k_diffusionsampling, "sample{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, extra_options) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "D:\gitProject\ComfyUI\comfy\k_diffusion\sampling.py", line 137, in sample_euler denoised = model(x, sigma_hat * s_in, extra_args) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 323, in forward out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "D:\gitProject\ComfyUI\comfy\k_diffusion\external.py", line 180, in forward return self.get_v(input * c_in, self.sigma_to_t(sigma), kwargs) c_out + input c_skip File "D:\gitProject\ComfyUI\comfy\samplers.py", line 302, in get_v return self.inner_model.apply_model(x, t, cond, kwargs) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 311, in apply_model out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 289, in sampling_function cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options) File "D:\gitProject\ComfyUI\comfy\samplers.py", line 265, in calc_cond_uncond_batch output = model_function(inputx, timestep, c).chunk(batch_chunks) File "D:\gitProject\ComfyUI\comfy\model_base.py", line 63, in apply_model return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float() File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "D:\gitProject\ComfyUI\custom_nodes\Comfyui-StableSR-main\modules\stablesr.py", line 157, in unet_forward self.struct_cond = self.struct_cond_model(self.latent_image, timesteps[:self.latent_image.shape[0]]) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "D:\gitProject\ComfyUI\custom_nodes\Comfyui-StableSR-main\modules\struct_cond.py", line 274, in forward emb = self.time_embed(timestep_embedding(timesteps, self.model_channels)) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward input = module(input) File "D:\gitProject\ComfyUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "D:\gitProject\ComfyUI\comfy\ops.py", line 18, in forward return torch.nn.functional.linear(input, self.weight, self.bias)

I hope you could tell me where I might have gone wrong and how I can get it to work properly. I'd really appreciate your help. Thanks in advance!

[xiongbb]

Error occurred when executing StableSRUpscalerPipe:

mat1 and mat2 must have the same dtype

File "E:\ComfyUI\ComfyUI\execution.py", line 153, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\ComfyUI\ComfyUI\execution.py", line 83, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\ComfyUI\ComfyUI\execution.py", line 76, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "E:\ComfyUI\ComfyUI\custom_nodes\Comfyui-StableSR\nodes.py", line 72, in doit_pipe upscale_image, color_map_image = upscaler.sample(image) File "E:\ComfyUI\ComfyUI\custom_nodes\Comfyui-StableSR\modules\stablesr.py", line 268, in sample nodes.common_ksampler(sd_model, seed, steps, cfg, sampler_name, scheduler, positive, negative, empty_latent, denoise)[0] File "E:\ComfyUI\ComfyUI\nodes.py", line 1207, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "E:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 22, in informative_sample raise e File "E:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample return original_sample(args, kwargs) File "E:\ComfyUI\ComfyUI\comfy\sample.py", line 97, in sample samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 785, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler(), sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 690, in sample samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 630, in sample samples = getattr(k_diffusionsampling, "sample{}".format(sampler_name))(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, extra_options) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "E:\ComfyUI\ComfyUI\comfy\k_diffusion\sampling.py", line 137, in sample_euler denoised = model(x, sigma_hat * s_in, extra_args) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 323, in forward out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, cond_concat=cond_concat, model_options=model_options, seed=seed) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\ComfyUI\ComfyUI\comfy\k_diffusion\external.py", line 180, in forward return self.get_v(input * c_in, self.sigma_to_t(sigma), kwargs) c_out + input c_skip File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 302, in get_v return self.inner_model.apply_model(x, t, cond, kwargs) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 311, in apply_model out = sampling_function(self.inner_model.apply_model, x, timestep, uncond, cond, cond_scale, cond_concat, model_options=model_options, seed=seed) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 289, in sampling_function cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options) File "E:\ComfyUI\ComfyUI\comfy\samplers.py", line 265, in calc_cond_uncond_batch output = model_function(inputx, timestep, c).chunk(batch_chunks) File "E:\ComfyUI\ComfyUI\comfy\model_base.py", line 63, in apply_model return self.diffusion_model(xc, t, context=context, y=c_adm, control=control, transformer_options=transformer_options).float() File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\ComfyUI\ComfyUI\custom_nodes\Comfyui-StableSR\modules\stablesr.py", line 157, in unet_forward self.struct_cond = self.struct_cond_model(self.latent_image, timesteps[:self.latent_image.shape[0]]) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "E:\ComfyUI\ComfyUI\custom_nodes\Comfyui-StableSR\modules\struct_cond.py", line 274, in forward emb = self.time_embed(timestep_embedding(timesteps, self.model_channels)) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\container.py", line 217, in forward input = module(input) File "E:\ComfyUI\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\ComfyUI\ComfyUI\comfy\ops.py", line 18, in forward return torch.nn.functional.linear(input, self.weight, self.bias)

[WSJUSA]

Hi guys, yes, this is THE issue, which is referenced on the readme page.

As I am not a Diffusion model expert unfortunately this configuration issue inside ConflyUI is beyond my knowledge. I have reached out to Light Chaser (pkuliyi2015) to try and get help on this, but thus far have not received any response. The issue is data types inside the instantiated diffusion are not aligned and its not enough to just force casting them to 16 (i've tried).

For an expert this is perhaps not too big an issue to resolve. However an expert has to be located to help for now.

I will update the read me to be more explicate

[gameltb]

diff --git a/modules/stablesr.py b/modules/stablesr.py
index c894055..e86398f 100644
--- a/modules/stablesr.py
+++ b/modules/stablesr.py
@@ -80,8 +80,8 @@ class StableSR:
         self.spade_layers.load_from_dict(state_dict)
         del state_dict

-        self.struct_cond_model.apply(lambda x: x.to(dtype=dtype, device=device))
-        self.spade_layers.apply(lambda x: x.to(dtype=dtype, device=device))
+        self.struct_cond_model.apply(lambda x: x.to(dtype=torch.float32, device=device))
+        self.spade_layers.apply(lambda x: x.to(dtype=torch.float32, device=device))
         self.latent_image: Tensor = None
         self.set_image_hooks = {}
         self.struct_cond: Tensor = None

I fixed the error with the patch above, but the result is not good

[Jeff-goal]

Thanks to @gameltb , I ran the stablesr node in comfyUI without bug. However, ... the left was original pic, while the right was after processed with comfyUI-stablesr, T T.

I hope this problem solved soon, as I have expected comfyUI-stablesr for a really long time. Stablesr is the best and irreplaceable in upscaling realphotos of lowres or blurry.

[WSJUSA]

As a practice messing with the Nvidia spade library is probably a bad idea.

That hack however probably points to something related in the unet configuration to address there.

The Unet implementation in ComfyUI is slightly different than the one in Auto1111.

Its way above my skill level since I don't know the details of what goes on inside. My hope is to finally attract attention of someone with the skills to understand this.

On Thu, Oct 19, 2023, 8:55 PM Jeff-goal @.***> wrote:

Thanks to @gameltb https://github.com/gameltb , I ran the stablesr node in comfyUI without bug. However, ... the left was original pic, while the right was after processed with comfyUI-stablesr, T T. [image: comfyUI-test] https://user-images.githubusercontent.com/148507731/276786805-3905ac55-3f3f-4784-be5a-4c5530c8ee50.png

I hope this problem solved soon, as I have expected comfyUI-stablesr for a really long time. Stablesr is the best and irreplaceable in upscaling realphotos of lowres or blurry.

— Reply to this email directly, view it on GitHub https://github.com/WSJUSA/Comfyui-StableSR/issues/1#issuecomment-1771903851, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQORMNYEWMKDBGLCKXZI5V3YAHDZFAVCNFSM6AAAAAA5XTDX3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRHEYDGOBVGE . You are receiving this because you commented.Message ID: @.***>

[gameltb]

another "hack" ,

diff --git a/modules/stablesr.py b/modules/stablesr.py
index c894055..100200f 100644
--- a/modules/stablesr.py
+++ b/modules/stablesr.py
@@ -226,7 +226,7 @@ class StableSRScript():
         y = pixels.shape[2]
         if pixels.shape[1] != x or pixels.shape[2] != y:
             pixels = pixels[:, :x, :y, :]
-        t = vae.encode(pixels[:, :, :, :3])
+        t = vae.encode(pixels[:, :, :, :3]) * 0.18215
         return {"samples": t}

     # sampler wrapper

have fun. @Jeff-goal

[xiongbb]

Thanks

[Jeff-goal]

 * 0.18215

Thanks a lot, awesome job! It was much better!

[tanis2010]

use this node to load model with a config file. It can generate correct image by t2i. But your i2i sampler can not work

[thekitchenscientist]

You can run comfyUI in 32bit mode adding the flag to the run_gpu.bat: --force-fp32

It runs now but the images are a garbled mess