search

1038lab / ComfyUI-OmniGen

ComfyUI-OmniGen - A ComfyUI custom node implementation of OmniGen, a powerful text-to-image generation and editing model.
MIT License
120 stars 11 forks source link

Phi3Transformer does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. #5

Closed tpolson closed 2 weeks ago

tpolson commented 2 weeks ago

ComfyUI Error Report

Error Details

1038lab commented 2 weeks ago

I've updated the custom node to support the eager attention implementation, so simply updating the custom node in comfyui-manager will resolve the issue.

This issue occurred due to using older versions of Python, PyTorch, or CUDA that don’t support the newer scaled_dot_product_attention (SDPA). With the eager implementation now in place, your setup should work without further issues.

For optimal performance and to fully benefit from SDPA, I recommend updating your Python, PyTorch, and CUDA versions when possible.

displague commented 2 weeks ago

Also received this on 7416002

Let me know what other details would be helpful:

!!! Exception during processing !!! Phi3Transformer does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument `attn_implementation="eager"` meanwhile. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")`
Traceback (most recent call last):
  File "/home/marques/src/ComfyUI/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)
  File "/home/marques/src/ComfyUI/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/ailab_OmniGen.py", line 217, in generation
    raise e
  File "/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/ailab_OmniGen.py", line 180, in generation
    pipe = self.OmniGenPipeline.from_pretrained(Paths.OMNIGEN_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/pipeline.py", line 82, in from_pretrained
    model = OmniGen.from_pretrained(model_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/model.py", line 197, in from_pretrained
    model = cls(config)
            ^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/model.py", line 186, in __init__
    self.llm = Phi3Transformer(config=transformer_config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.py", line 948, in __init__
    super().__init__(config)
  File "/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1388, in __init__
    config = self._autoset_attn_implementation(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1565, in _autoset_attn_implementation
    config = cls._check_and_enable_sdpa(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1731, in _check_and_enable_sdpa
    raise ValueError(
ValueError: Phi3Transformer does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument `attn_implementation="eager"` meanwhile. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")`

Very GPU/VRAM poor, but I've got some RAM.

Total VRAM 4034 MB, total RAM 62897 MB
pytorch version: 2.4.1+cu121
xformers version: 0.0.28.post1
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1050 Ti with Max-Q Design : cudaMallocAsync
tpolson commented 2 weeks ago

The fix ended being updating torch. 

On Nov 8, 2024, 9:18 AM, at 9:18 AM, Marques Johansson @.***> wrote:

Also received this on 7416002

Let me know what other details would be helpful:

!!! Exception during processing !!! Phi3Transformer does not support an
attention implementation through
torch.nn.functional.scaled_dot_product_attention yet. Please request
the support for this architecture:
https://github.com/huggingface/transformers/issues/28005. If you
believe this error is a bug, please open an issue in Transformers
GitHub repository and load your model with the argument
`attn_implementation="eager"` meanwhile. Example: `model =
AutoModel.from_pretrained("openai/whisper-tiny",
attn_implementation="eager")`
Traceback (most recent call last):
File "/home/marques/src/ComfyUI/execution.py", line 323, in
execute
output_data, output_ui, has_subgraph = get_output_data(obj,
input_data_all, execution_block_cb=execution_block_cb,
pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/marques/src/ComfyUI/execution.py", line 198, in
get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION,
allow_interrupt=True, execution_block_cb=execution_block_cb,
pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/marques/src/ComfyUI/execution.py", line 169, in
_map_node_over_list
   process_inputs(input_dict, i)
File "/home/marques/src/ComfyUI/execution.py", line 158, in
process_inputs
   results.append(getattr(obj, func)(**inputs))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/ailab_OmniGen.py",
line 217, in generation
   raise e
File
"/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/ailab_OmniGen.py",
line 180, in generation
   pipe = self.OmniGenPipeline.from_pretrained(Paths.OMNIGEN_DIR)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/pipeline.py",
line 82, in from_pretrained
   model = OmniGen.from_pretrained(model_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/model.py",
line 197, in from_pretrained
   model = cls(config)
           ^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/custom_nodes/ComfyUI-OmniGen/OmniGen/model.py",
line 186, in __init__
   self.llm = Phi3Transformer(config=transformer_config)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.py",
line 948, in __init__
   super().__init__(config)
File
"/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py",
line 1388, in __init__
   config = self._autoset_attn_implementation(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py",
line 1565, in _autoset_attn_implementation
   config = cls._check_and_enable_sdpa(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/marques/src/ComfyUI/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py",
line 1731, in _check_and_enable_sdpa
   raise ValueError(
ValueError: Phi3Transformer does not support an attention
implementation through torch.nn.functional.scaled_dot_product_attention
yet. Please request the support for this architecture:
https://github.com/huggingface/transformers/issues/28005. If you
believe this error is a bug, please open an issue in Transformers
GitHub repository and load your model with the argument
`attn_implementation="eager"` meanwhile. Example: `model =
AutoModel.from_pretrained("openai/whisper-tiny",
attn_implementation="eager")`

Very GPU/VRAM poor, but I've got some RAM.

Total VRAM 4034 MB, total RAM 62897 MB
pytorch version: 2.4.1+cu121
xformers version: 0.0.28.post1
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce GTX 1050 Ti with Max-Q Design :
cudaMallocAsync

-- Reply to this email directly or view it on GitHub: https://github.com/1038lab/ComfyUI-OmniGen/issues/5#issuecomment-2465322169 You are receiving this because you authored the thread.

Message ID: @.***>

displague commented 2 weeks ago

I updated to torch 2.5.1 (using the first command below) and got the same error. I ran a few more phases of updates and it seems to be moving along now.

# within ComfyUI; after source .venv/bin/activate
pip install --upgrade torch{,vision,audio,scale}
pip install --upgrade gradio pillow mediapipe protobuf grpcio-status
pip install --upgrade -r requirements.txt
tpolson commented 2 weeks ago

I just ran the dependencies batch in the comfui updates folder.  Now i just need 48gb of vram. Lol

On Nov 8, 2024, 3:18 PM, at 3:18 PM, Marques Johansson @.***> wrote:

I updated to torch 2.5.1 (using the first command below) and got the same error. I ran a few more phases of updates and it seems to be moving along now.

# within ComfyUI; after source .venv/bin/activate
pip install --upgrade torch{,vision,audio,scale}
pip install --upgrade gradio pillow mediapipe protobuf grpcio-status
pip install --upgrade -r requirements.txt

-- Reply to this email directly or view it on GitHub: https://github.com/1038lab/ComfyUI-OmniGen/issues/5#issuecomment-2465889699 You are receiving this because you authored the thread.

Message ID: @.***>

displague commented 2 weeks ago

Yeah, no luck here either. I also tried with --cpu and the 64G still oom'd.

tpolson commented 2 weeks ago

I'm getting good results with 24gb with high settings at low res, but i want to match depthanythingv2 quality at 4k.

On Nov 8, 2024, 5:29 PM, at 5:29 PM, Marques Johansson @.***> wrote:

Yeah, no luck here either. I also tried with --cpu and the 64G still oom'd.

-- Reply to this email directly or view it on GitHub: https://github.com/1038lab/ComfyUI-OmniGen/issues/5#issuecomment-2465969548 You are receiving this because you authored the thread.

Message ID: @.***>