search

comfyanonymous / ComfyUI_TensorRT

MIT License
513 stars 34 forks source link

Error occurred when executing Flux model conversion #65

Open deepfree2023 opened 3 months ago

deepfree2023 commented 3 months ago

All flux models, dev, schnell and fp8 versions, report this error during conversion, whether use Dynamic or Static conversion:

File "K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1543, in _slow_forward result = self.forward(*input, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "K:\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\layers.py", line 166, in forward attn = attention(torch.cat((txt_q, img_q), dim=2), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "K:\ComfyUI_windows_portable\ComfyUI\comfy\ldm\flux\math.py", line 11, in attention x = optimized_attention(q, k, v, heads, skip_reshape=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "K:\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py", line 361, in attention_xformers return attention_pytorch(q, k, v, heads, mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "K:\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py", line 400, in attentionpytorch b, , dim_head = q.shape ^^^^^^^^^^^^^^ ValueError: too many values to unpack (expected 3)

deepfree2023 commented 3 months ago

CUDA 12.1, torch 2.4.0, tensorrt 10.3.0.26

XmYx commented 3 months ago

i could overcome this by adding return attention_pytorch(q, k, v, heads, mask, skip_reshape=skip_reshape) at line 361 of attention.py, as that's not passed further, and causing this mishap. Then you'll face some other issue, and will need over 24Gb of VRAM to succeed, it failed on my 4090. also disabling xformers might help(?). I noticed, with it disabled, torch compile's take a little less vram.

wogam commented 3 months ago

Currently there's issue with bf16 so it doesn't work: https://huggingface.co/black-forest-labs/FLUX.1-dev/discussions/24