cuDNN Frontend error: [cudnn_frontend] Error: No execution plans support the graph.

[dobby-hsb]

I'm getting a cuDNN Frontend error and I can't seem to resolve it.

ComfyUI Error Report

Error Details

• Node Type: PyramidFlowTextEncode
• Exception Type: RuntimeError

• Exception Message: cuDNN Frontend error: [cudnn_frontend] Error: No execution plans support the graph.

  Stack Trace

  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 323, in execute
  output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 198, in get_output_data
  return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 169, in _map_node_over_list
  process_inputs(input_dict, i)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 158, in process_inputs
  results.append(getattr(obj, func)(**inputs))
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/nodes.py", line 264, in sample
  prompt_embeds, prompt_attention_mask, pooled_prompt_embeds = text_encoder(positive_prompt, device)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 138, in forward
  prompt_embeds, prompt_attention_mask, pooled_prompt_embeds = self.encode_prompt(input_prompts, 1, clip_skip=None, device=device)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 113, in encode_prompt
  pooled_prompt_embed = self._get_clip_prompt_embeds(
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 98, in _get_clip_prompt_embeds
  prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1413, in forward
  text_outputs = self.text_model(
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 901, in forward
  encoder_outputs = self.encoder(
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 824, in forward
  layer_outputs = encoder_layer(
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 559, in forward
  hidden_states, attn_weights = self.self_attn(
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
  return forward_call(*args, **kwargs)
  
  File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 491, in forward
  attn_output = torch.nn.functional.scaled_dot_product_attention(

## System Information
- **ComfyUI Version:** v0.2.3-17-gce0542e
- **Arguments:** /projects/stability_matrix/Data/Packages/ComfyUI/main.py --port 8189 --preview-method auto
- **OS:** posix
- **Python Version:** 3.10.11 (main, May  7 2023, 19:26:31) [Clang 16.0.3 ]
- **Embedded Python:** false
- **PyTorch Version:** 2.5.0+cu121
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
  - **Type:** cuda
  - **VRAM Total:** 25756696576
  - **VRAM Free:** 12796621616
  - **Torch VRAM Total:** 11207180288
  - **Torch VRAM Free:** 12383024

## Logs

2024-10-18 18:11:20,538 - root - ERROR - Traceback (most recent call last): File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 323, in execute output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 198, in get_output_data return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb) File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 169, in _map_node_over_list process_inputs(input_dict, i) File "/projects/stability_matrix/Data/Packages/ComfyUI/execution.py", line 158, in process_inputs results.append(getattr(obj, func)(inputs)) File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/nodes.py", line 264, in sample prompt_embeds, prompt_attention_mask, pooled_prompt_embeds = text_encoder(positive_prompt, device) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 138, in forward prompt_embeds, prompt_attention_mask, pooled_prompt_embeds = self.encode_prompt(input_prompts, 1, clip_skip=None, device=device) File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 113, in encode_prompt pooled_prompt_embed = self._get_clip_prompt_embeds( File "/projects/stability_matrix/Data/Packages/ComfyUI/custom_nodes/ComfyUI-PyramidFlowWrapper/pyramid_dit/modeling_text_encoder.py", line 98, in _get_clip_prompt_embeds prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 1413, in forward text_outputs = self.text_model( File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 901, in forward encoder_outputs = self.encoder( File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 824, in forward layer_outputs = encoder_layer( File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 559, in forward hidden_states, attn_weights = self.self_attn( File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) File "/projects/stability_matrix/Data/Packages/ComfyUI/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 491, in forward attn_output = torch.nn.functional.scaled_dot_product_attention( RuntimeError: cuDNN Frontend error: [cudnn_frontend] Error: No execution plans support the graph.

[techniqs]

I fixed this problem via downgrading pytorch from 2.5.0 to 2.4.1 looks like its something about pytorch breaking SDPA where i got it from