yuvraj108c / ComfyUI-Upscaler-Tensorrt

3-4x faster ComfyUI Image Upscaling using Tensorrt
Other
61 stars 4 forks source link

Node execution crash #13

Closed rustyblood closed 1 month ago

rustyblood commented 3 months ago

When using the node and feeding about 4 images coming from a prevous node (image batch), the upscaling process is working fine. It takes about 2-4 seconds to start and it finishes in about another 5 seconds tops. When feeding more images into another upscaler node, about 20-40 images, it worked properly for about 2 days, then it stoped working. I have deleted the node and insert a new one, but when the node execution begins, I can see the line in the command line window stating that it is loading the Tensorrt engine and it stops, even locking the whole computer. I need to close the terminal window and I can resume activities. Engine was regenerated 2 times using 2 different onnx models (4x foolhardy and 4x AnimeSharp) with same results.

image

yuvraj108c commented 3 months ago

Hi, it might be a vram/ram issue, monitor your resources and check if anything is maxing out

rustyblood commented 3 months ago

Hi! I have tried it again. I am not able to see if there is anything maxing out, since at the very beginning of the process (when tensorrt node is starting), the resource monitor stops also. I see a normal increase in VRAM (RTX4080, 12Gb) only, but RAM (32Gb) and CPU (13900HX) are normal. After letting it run for a bout 5 mins or so, it keeps going for a very brief moment and it locks the computer again. The resource monitor does advance, showing only 11Gb of VRAM being used and RAM / CPU normal use (40% / 20%). Curious thing is that when I installed the node the first time, it worked flawlesly, incredibly fast with the same configuration, but not it does not LOL. I will try with a lower image batch number and see how it reacts, maybe it is just too many images for the amount of VRAM available?

EDIT: After trying several variations, the node crashes with as little as 10 images input. It makes the computer go very very slow but I am able to close the terminal window and regain control.

yuvraj108c commented 3 months ago

The last code update I made was 1 month ago, so you have to assess where the issue is coming from.

Try to use an empty workflow with only this node and upscale 10 images & check if the issues persist.

rustyblood commented 3 months ago

Thank you !! I made a new workflow and tryed 5, 10, 20, 30 and 40 images (basic generation: checkpoint, prompt, sampler, decode, preview, upscale). It ran perfectly for the first run of each set. After trying a second round of 40 images, it stopped at the beginning of the upscale node. Trying a new rerun, process started to run incredibly slow around half of the process and it stops. Restarting comfyui does not seem to help. Restarting the whole computer allows to re-run the workflow for 1 time, but on the first rerun, it stops again. Seems that the process is tryig to get as much resources as possible, but after finishing the first run, they might not be freed for the next run. image

yuvraj108c commented 3 months ago

ok, I'll take a look on the issue thanks for reporting in details

rustyblood commented 3 months ago

No problem, thank you for this module. It works really fast, I mean, blazing fast compared with the regular upscaling process (50% o more faster). It helps on reducing time inmensely.

rustyblood commented 3 months ago

I tried to get the node again on a new comfyui reinstall from scratch and a new workflow using a 1024 x 640 image resolution, so it should be still within the max 1280 x 1280 resolution. It still works flawlessly on the first run, but starting the second run, it simply halts. Even restarting Comfyui, the node does not start, but restarting the computer allows for a 1 time run, next run halts the computer again.

yuvraj108c commented 3 months ago

It is most likely a vram problem. 12gb is not enough based on my tests.

rustyblood commented 3 months ago

Got it, thank you for all your support. Last question, since the node works at least 1 time with 12Gb with no problems, how can the Vram be "freed" for the next run, allowing for a possible next operation?

yuvraj108c commented 1 month ago

Try using a batch manager for upscaling long videos e.g batch manager from videohelpersuite