search

aszc-dev / ComfyUI-CoreMLSuite

A set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows.
GNU General Public License v3.0
92 stars 8 forks source link

Error when trying to sample with converted SD1.5 inpainting models: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1 #44

Open lquesada opened 1 month ago

lquesada commented 1 month ago

The example workflow works for me using e.g. using a converted dreamshaper_8.

However, if I convert an inpainting model (tried several, but e.g. sd-v1-5-inpainting.ckpt) to Core ML, I try to run the example workflow with it, the Core ML Sampler errors (see error 1 below). I've tried CPU_AND_NE, CPU_AND_GPU.

I think the example workflow automatically creates a latent the right size. However if I explicitly plug in an "empty latent image" with the right size (512x512, as the model was converted and double checked with the model name", the sampler errors as well (see error 2 below)

I believe the two errors may be related to the inpainting model, as using "empty latent image" with the right size with dreamshaper_8 works well.

Are inpainting models not supported? Any way to work it around? Is it an issue with this model in particular?

error 1:

Error occurred when executing CoreMLSampler:

The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

Traceback (most recent call last): File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample return super().sample( File "/Users/user/ComfyUI/nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample return orig_comfy_sample(model, *args, *kwargs) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample return orig_comfy_sample(model, args, kwargs) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 22, in informative_sample raise e File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample return original_sample(args, kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, self.extra_options) File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m denoised = model(x, sigmas[i] * s_in, *extra_args) File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call return self.predict_noise(args, kwargs) File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch output = model.apply_model(inputx, timestep, *c).chunk(batch_chunks) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper return orig_apply_model(self, args, *kwargs) File "/Users/user/ComfyUI/comfy/model_base.py", line 98, in apply_model return self.model_sampling.calculate_denoised(sigma, model_output, x) File "/Users/user/ComfyUI/comfy/model_sampling.py", line 12, in calculate_denoised return model_input - model_output sigma RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 1

Prompt executed in 1.67 seconds

error 2:

Error occurred when executing CoreMLSampler:

Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample Traceback (most recent call last): File "/Users/user/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/Users/user/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/Users/user/ComfyUI/execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/nodes.py", line 65, in sample return super().sample( File "/Users/user/ComfyUI/nodes.py", line 1344, in sample return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise) File "/Users/user/ComfyUI/nodes.py", line 1314, in common_ksampler samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample return orig_comfy_sample(model, *args, *kwargs) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 111, in uncond_multiplier_check_cn_sample return orig_comfy_sample(model, args, kwargs) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample return original_sample(args, kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations. File "/Users/user/ComfyUI/comfy/sample.py", line 37, in sample samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 761, in sample return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 663, in sample return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 650, in sample output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 629, in inner_sample samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar) File "/Users/user/ComfyUI/comfy/samplers.py", line 534, in sample samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, self.extra_options) File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/Users/user/ComfyUI/comfy/k_diffusion/sampling.py", line 583, in sample_dpmpp_2m denoised = model(x, sigmas[i] * s_in, *extra_args) File "/Users/user/ComfyUI/comfy/samplers.py", line 272, in call out = self.inner_model(x, sigma, model_options=model_options, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 616, in call return self.predict_noise(args, kwargs) File "/Users/user/ComfyUI/comfy/samplers.py", line 619, in predict_noise return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed) File "/Users/user/ComfyUI/comfy/samplers.py", line 258, in sampling_function out = calc_cond_batch(model, conds, x, timestep, model_options) File "/Users/user/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch output = model.apply_model(inputx, timestep, *c).chunk(batch_chunks) File "/Users/user/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 63, in apply_model_uncond_cleanup_wrapper return orig_apply_model(self, args, kwargs) File "/Users/user/ComfyUI/comfy/model_base.py", line 97, in apply_model model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, extra_conds).float() File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 23, in call chunked_out = [ File "/Users/user/ComfyUI/custom_nodes/ComfyUI-CoreMLSuite/coreml_suite/models.py", line 25, in self.coreml_model(input_kwargs.coreml_kwargs(self.expected_inputs)), File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 97, in call self._verify_inputs(kwargs) File "/Users/user/ComfyUI/venv/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py", line 90, in _verify_inputs raise TypeError( TypeError: Expected shape (1, 9, 64, 64), got (1, 4, 64, 64) for input: sample

Prompt executed in 0.06 seconds

aszc-dev commented 1 week ago

Hello, @lquesada,

Interesting point. Indeed, I haven’t tested the inpainting models. As they work differently internally compared to the regular models, I’ll see if adding support for them is feasible. Until then, it seems like using inpaint checkpoints is not possible.

Cheers, Adrian