RAM overflow in cycles (maybe problems with Execution Model Inversion)

[Shyryp]

Expected Behavior

I assumed that ForLoop loops do not cause RAM overflows. Or I expected to find a built-in way to controllably clear memory inside loops (inside the ForLoop body and after the ForLoop function has finished working).

Actual Behavior

At the moment, even with simple image tools (and maybe others data, I haven't checked yet), memory overflow occurs, since RAM is not cleared of images created in a loop. In this case, after the cycle is completed, the data is stored in memory until the current task (Queue prompt) in the queue is completed.

I used custom nodes with while/forloop functions by @guill (@BadCafeCode) from https://github.com/BadCafeCode/execution-inversion-demo-comfyui

I checked everything on the latest versions of ComfyUI and on the latest versions of custom node packages. Just a few minutes ago - the memory overflow problem exists.

Memory overflow occurs in any node that somehow works with images and modifies them.

Issue/bug for @guill

Steps to Reproduce

As an example, I made a simple workflow in which a blur operation is performed in a loop for the same image. Over time, memory overflow occurs (if this does not happen for you, set a higher value instead of 1000 for remaining): Overflow_Example_Workprocess.json

Screenshot of workprocess:

Debug Logs

I didn't record logs, but if necessary - you can ask me.

When memory is full, comfyui just crashes without any errors.

Other

Related topics: https://github.com/BadCafeCode/execution-inversion-demo-comfyui/issues/8 https://github.com/lquesada/ComfyUI-Inpaint-CropAndStitch/issues/32

The problem probably cannot be in custom nodes or any other nodes (including ComfyCore nodes), since they do not clean up memory by themselves during the execution of loops.

Probably, some manually controlled tool (node) for cleaning up memory in the loop, or a mechanism that will clean up memory inside the loop will be required.

P.s. Perhaps I am not aware of some features and nodes from the execution-inversion-demo-comfyui package, and perhaps there are already some ways to clear memory, but I do not know about them / did not find them in the examples. The cleaning systems that I know do not affect the cleaning of memory with images during the loop.

[guill]

The fundamental issue here is that ephemeral nodes created during execution are cached for the next run just like any other node -- you would see this same issue if you manually copy+pasted the Blur node 1000 times.

To solve this, we need to be able to start removing node outputs from the cache while execution is still occurring (but after we're sure no other node will use the node's outputs).

[Shyryp]

Another addition from me: Of course, I'm not going to use loops with repetitions of thousands of times. But the problem easily arises even with small amounts of repetition - sometimes ten repetitions are enough with average complexity of nodes inside the loop, after which the workflow catches a memory overflow and crashes ComfyUI.

This problem especially arises very quickly with any high-resolution images inside the loop and any manipulations with these images inside the loop.

Such an example is my workflow from this link, which uses the Inpaint Crop node: https://github.com/lquesada/ComfyUI-Inpaint-CropAndStitch/issues/32

That is, if the loop contains 10 more or less memory-intensive editors of large images, then the number of permissible cycles before the overflow is reduced to units.

[ecpknymt]

Yes, when Comfyui memory overflow, it just crush, even if I click free model and node cache button, the server crash.

Maybe a node free ram with a threshold, I don't know...

[BrechtCorbeel]

Most people using python aren't used to or trained to clear memory because python does that for you unlike other languages, so I get where it comes from, but it is an issue.