search

TemryL / ComfyUI-IDM-VTON

ComfyUI adaptation of IDM-VTON for virtual try-on.
GNU General Public License v3.0
370 stars 40 forks source link

out of memory issue #4

Open TemryL opened 5 months ago

TemryL commented 5 months ago

should fix: RuntimeError: MPS backend out of memory (MPS allocated: 16.56 GB, other allocations: 128.70 MB, max allowed: 18.13 GB). Tried to allocate 3.00 GB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

same issue on cuda with small gpu

marcojoao commented 5 months ago

also got out of ram:

`Error occurred when executing PipelineLoader:

Allocation on device 0 would exceed allowed memory. (out of memory) Currently allocated : 2.44 GiB Requested : 5.00 MiB Device limit : 22.19 GiB Free (according to CUDA): 6.50 MiB PyTorch limit (set by user-supplied memory fraction) : 17179869184.00 GiB

File "/home/runner/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ComfyUI/execution.py", line 69, in map_node_over_list results.append(getattr(obj, func)()) ^^^^^^^^^^^^^^^^^^^^ File "/home/runner/ComfyUI/custom_nodes/ComfyUI-IDM-VTON/src/nodes/pipeline_loader.py", line 46, in load_pipeline ).requiresgrad(False).eval().to(DEVICE) ^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1152, in to return self._apply(convert) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 802, in _apply module._apply(fn) File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 802, in _apply module._apply(fn) File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 802, in _apply module._apply(fn) [Previous line repeated 6 more times] File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 825, in _apply param_applied = fn(param) ^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1150, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^`

TemryL commented 5 months ago

Yes it is a known issue that needs to be fixed, I only managed to make it work on gpu with 32gb+ of vram. Don't hesitate to make PRs if you find a way to fix it :)

marcojoao commented 5 months ago

Yes it is a known issue that needs to be fixed, I only managed to make it work on gpu with 32gb+ of vram. Don't hesitate to make PRs if you find a way to fix it :)

sure I'll take a look

pranay0318 commented 5 months ago

Is there a way we can solve out of memory issue now?

pranay0318 commented 5 months ago

Here's the Error I got!

Some weights of the model checkpoint were not used when initializing UNet2DConditionModel: ['add_embedding.linear_1.bias, add_embedding.linear_1.weight, add_embedding.linear_2.bias, add_embedding.linear_2.weight'] !!! Exception during processing!!! Allocation on device Traceback (most recent call last): File "/home/ubuntu/Comfy/ComfyUI/execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/home/ubuntu/Comfy/ComfyUI/execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/home/ubuntu/Comfy/ComfyUI/execution.py", line 69, in map_node_over_list results.append(getattr(obj, func)()) File "/home/ubuntu/Comfy/ComfyUI/custom_nodes/ComfyUI-IDM-VTON/src/nodes/pipeline_loader.py", line 58, in load_pipeline ).requiresgrad(False).eval().to(DEVICE) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1173, in to return self._apply(convert) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply module._apply(fn) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply module._apply(fn) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply module._apply(fn) [Previous line repeated 4 more times] File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 804, in _apply param_applied = fn(param) File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1159, in convert return t.to( torch.cuda.OutOfMemoryError: Allocation on device

SuperMasterBlasterLaser commented 4 months ago

I've encountered this error too.

When I launch your nodes with your example. It works good. It uses preloaded image and applies garment.

The problem is when I add SD Checkpoint and try to send generated image into Run IDM-VTON Inference node, where it throws this error.

My Specs:

  1. RAM - 32GB
  2. VRAM (GPU) - 24GB

I think we need some kind of memory unloading?

HashedViking commented 3 months ago

Yes it is a known issue that needs to be fixed, I only managed to make it work on gpu with 32gb+ of vram. Don't hesitate to make PRs if you find a way to fix it :)

Hi, @TemryL , PipelineLoader node crashes ComfyUI no matter what I choose f32/f16/b My spec: VRAM 24 Gb, RAM 16 GB, wsl2 Have you managed to lower VRAM requirements? Any advice on how to debug this crash would be really appreciated. Thank you in advance.

HashedViking commented 3 months ago

Yes it is a known issue that needs to be fixed, I only managed to make it work on gpu with 32gb+ of vram. Don't hesitate to make PRs if you find a way to fix it :)

Hi, @TemryL , PipelineLoader node crashes ComfyUI no matter what I choose f32/f16/b My spec: VRAM 24 Gb, RAM 16 GB, wsl2 Have you managed to lower VRAM requirements? Any advice on how to debug this crash would be really appreciated. Thank you in advance.

fixed by adding 16 GB more RAM to the wsl, so 24 GB VRAM + 32 GB RAM seems enough

kunkun-zhu commented 3 months ago

是的，这是一个需要修复的已知问题，我只能让它在具有 32gb+ vram 的 gpu 上工作。如果你找到修复它的方法，不要犹豫，做 PR :)

你好，@TemryL，无论我选择 f32/f16/b，PipelineLoader 节点都会使 ComfyUI 崩溃。我的规格：VRAM 24 Gb、RAM 16 GB、wsl2 您是否已设法降低 VRAM 要求？任何有关如何调试此崩溃的建议都将不胜感激。提前谢谢您。

通过向 wsl 添加 16 GB 的 RAM 来修复此问题，因此 24 GB VRAM + 32 GB RAM 似乎足够了

Yes it is a known issue that needs to be fixed, I only managed to make it work on gpu with 32gb+ of vram. Don't hesitate to make PRs if you find a way to fix it :)

Hi, @TemryL , PipelineLoader node crashes ComfyUI no matter what I choose f32/f16/b My spec: VRAM 24 Gb, RAM 16 GB, wsl2 Have you managed to lower VRAM requirements? Any advice on how to debug this crash would be really appreciated. Thank you in advance.

fixed by adding 16 GB more RAM to the wsl, so 24 GB VRAM + 32 GB RAM seems enough

hello,My server also reported out of memory with 24 GB VRAM and 240 GB RAM. Are you sure you have resolved this issue？

HashedViking commented 3 months ago

@kunkun-zhu hi, I've only tried the example workflow.png. Maybe you have an advanced one that causes additional RAM/VRAM load. My server shows ~ 17 GB RAM and ~20 GB VRAM utilisation.