Add option to auto retry at out of memory runtime error as memory leak workaround

[T0biasCZe]

Currently there are some bugs that cause a Runtime error during batch conversion, because there are some memory leaks that cause the system to run out of memory after processing few hundred files.
But, when starting the conversion again after the error, it again starts processing fine until it again runs out of memory after some time.
So, until the memory leaks are fixed, as temporary workaround there could be an option to automatically retry when there is an memory error, so you wouldnt have to do it manually...

[graemeniedermayer]

Could you graph the leak (screenshot a task managers memory graph)? From the description it sounds like it might be images not being offloaded from memory during generation. This probably will be a limitation for large video processing.

[T0biasCZe]

2 seconds before the crash: (problem is that since the whole PC runs out of memory, OBS also runs out of memory and crashes the recording) after crash:

2h earlier before the crash:

30m before the crash:

there wasnt anything else running in background other than OBS that would eat up memory (and it crashes like this even without OBS)

[graemeniedermayer]

Great! So as a rough calculation 6gb for 6000 images seems like it's roughly 1mb/image which means the images aren't released from memory after generation.

So the likely culprit is a line like https://github.com/thygate/stable-diffusion-webui-depthmap-script/blob/5250c8b7ea895a4b15575d17a278c88c38cb2132/src/common_ui.py#L545

Where images are added to a list without being released. This line is necessary for displaying the images, but when generating thousands of images of video might become a problem. I'm not sure what the best solution would be. Like a toggle for displaying images or a cap for displaying images.

Quick Fix

you should be able to comment out this line (This will remove the image display functionality). So adding a # in front # img_results += [(input_i, type, result)] this line is located in src/common_ui.py. Also if this doesn't fix the leak let me know.

[semjon00]

Interesting!

[semjon00]

@graemeniedermayer you might look into if it is possible to add a general fix so it does not consume memory if there is no need to. Would be appreciated!