search

huchenlei / ComfyUI-layerdiffuse

Layer Diffuse custom nodes
Apache License 2.0
1.48k stars 145 forks source link

[Bug]: RuntimeError: "addmm_impl_cpu_" not implemented for 'Half' in the recent comfyui commits. #63

Open ethanfel opened 7 months ago

ethanfel commented 7 months ago

What happened?

RuntimeError: "addmm_implcpu" not implemented for 'Half' in the recent comfyui commits.

Steps to reproduce the problem

running wf

What should have happened?

The mm call (matrix multiplication) requires float32, but it gets a half type (float16)

Commit where the problem happens

ComfyUI: after d7897ff ComfyUI-layerdiffuse:

Sysinfo

Linux:RTX4090

Console logs

!!! Exception during processing !!!
Traceback (most recent call last):
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/efficiency-nodes-comfyui/efficiency_nodes.py", line 2175, in sample_adv
    return super().sample(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative,
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/efficiency-nodes-comfyui/efficiency_nodes.py", line 700, in sample
    samples, images, gifs, preview = process_latent_image(model, seed, steps, cfg, sampler_name, scheduler,
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/efficiency-nodes-comfyui/efficiency_nodes.py", line 537, in process_latent_image
    samples = KSamplerAdvanced().sample(model, add_noise, seed, steps, cfg, sampler_name, scheduler,
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/nodes.py", line 1403, in sample
    return common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/nodes.py", line 1339, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 22, in informative_sample
    raise e
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 267, in motion_sample
    return orig_comfy_sample(model, noise, *args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/sample.py", line 100, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI_smZNodes/smZNodes.py", line 1380, in KSampler_sample
    return _KSampler_sample(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 705, in sample
    return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI_smZNodes/smZNodes.py", line 1399, in sample
    return _sample(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 610, in sample
    samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 548, in sample
    samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/k_diffusion/sampling.py", line 613, in sample_dpmpp_2m_sde
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 286, in forward
    out = self.inner_model(x, sigma, cond=cond, uncond=uncond, cond_scale=cond_scale, model_options=model_options, seed=seed)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 273, in forward
    return self.apply_model(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI_smZNodes/smZNodes.py", line 1012, in apply_model
    out = super().apply_model(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 270, in apply_model
    out = sampling_function(self.inner_model, x, timestep, uncond, cond, cond_scale, model_options=model_options, seed=seed)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 250, in sampling_function
    cond_pred, uncond_pred = calc_cond_uncond_batch(model, cond, uncond_, x, timestep, model_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-TiledDiffusion/.patches.py", line 4, in calc_cond_uncond_batch
    return calc_cond_uncond_batch_original_tiled_diffusion_778392d4(model, cond, uncond, x_in, timestep, model_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/samplers.py", line 224, in calc_cond_uncond_batch
    output = model.apply_model(input_x, timestep_, **c).chunk(batch_chunks)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/model_base.py", line 96, in apply_model
    model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/SeargeSDXL/modules/custom_sdxl_ksampler.py", line 70, in new_unet_forward
    x0 = old_unet_forward(self, x, timesteps, context, y, control, transformer_options, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 850, in forward
    h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 44, in forward_timestep_embed
    x = layer(x, context, transformer_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/attention.py", line 633, in forward
    x = block(x, context=context[i], transformer_options=transformer_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-layerdiffuse/lib_layerdiffusion/attention_sharing.py", line 253, in forward
    return func(self, x, context, transformer_options)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/attention.py", line 460, in forward
    return checkpoint(self._forward, (x, context, transformer_options), self.parameters(), self.checkpoint)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 191, in checkpoint
    return func(*inputs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/comfy/ldm/modules/attention.py", line 520, in _forward
    n = self.attn1(n, context=context_attn1, value=value_attn1)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-layerdiffuse/lib_layerdiffusion/attention_sharing.py", line 212, in forward
    q = self.to_q_lora[f](modified_hidden_states[f])
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/p5/StabilityMatrix/Packages/ComfyUI/custom_nodes/ComfyUI-layerdiffuse/lib_layerdiffusion/attention_sharing.py", line 70, in forward
    final_weight = org_weight + torch.mm(up_weight, down_weight)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Workflow json file

none

Additional information

No response

huchenlei commented 7 months ago

Cannot verify. https://github.com/comfyanonymous/ComfyUI/commit/a28a9dc83684624ee2167c0b92d976bb68f2c606 seems to work fine.

ethanfel commented 7 months ago

Cannot verify. comfyanonymous/ComfyUI@a28a9dc seems to work fine.

Added a simple WF to illustrate the issue.

start comfyui -> run -> error.

but

bypass layer diff -> run (work) -> enable layer diff -> run (work until the model is changed or comfyui restarted)

error.json

ethanfel commented 7 months ago

the bug is still present in the later comfyui commit.

yueeng commented 2 months ago

Same error at: https://github.com/comfyanonymous/ComfyUI/commit/5d43e75e5b94c203075e315e4516fee47c4d6950