search

comfyanonymous / ComfyUI

The most powerful and modular stable diffusion GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
43.53k stars 4.61k forks source link

Upscale Image (using Model) crashes before sending output #3240

Open ricperry opened 3 months ago

ricperry commented 3 months ago

My batch image upscale is crashing just prior to sending the output to the next node. ComfyUI crashes without outputting any error messages. My usage situation follows:

140 frames 576 x 576 res 16bit pngs 4x_NMKD-Siax_200k.pth upscaler standalone workflow (no SD models loaded, just a plain pixel space upscaling workflow after a system reboot)

image

Terminal output:

got prompt [rgthree] Using rgthree's optimized recursive execution. [rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute. [rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI. got prompt got prompt got prompt ./comfy.sh: line 5: 8152 Killed python main.py --auto-launch --listen --fp32-vae

WilliamStanford commented 3 months ago

Getting the same problem, any luck solving it?

ltdrdata commented 3 months ago

Monitor to see if it's not a lack of memory.

WilliamStanford commented 3 months ago

Monitor to see if it's not a lack of memory.

That's what I figured it would be, only happens when trying to upscale many frames at once. Any tips on how to get around this?

I was thinking I could try split the images into batches and then recombine them, but I'm not quite sure how to do that using ComfyUI.

ricperry commented 3 months ago

Yes, it exceeded my system ram + swap, then the process just quits. My VRAM was only at ~60%. I guess I'll have to expand the size of my swap file. [update] That didn't fix it. There's a runaway memory spike at the end of the node. It consumed 8G swap space in a matter of seconds and quickly exceeded my 40G RAM (32G memory + 8G swap). Earlier I had been using only 2G swap, and it didn't ramp up on memory usage so quickly.

ltdrdata commented 3 months ago

In such cases, I recommend using a node that fetches images sequentially one by one from the folder, rather than fetching all at once, and processing them separately with an auto queue. From what I remember, there is such a node in the WAS node.

FYI, That issue is unrelated to VRAM. When the system RAM reaches a critical state, it is the OS that forces the shutdown.

AgeOfAlgorithms commented 2 months ago

I'm having the exact same issue but with video upscaling ksampler. Did anyone solve this problem?