n00mkrad / cupscale

Image Upscaling GUI based on ESRGAN
MIT License
2.17k stars 111 forks source link

Unable to upscale full image because of time out #101

Open RunDevelopment opened 2 years ago

RunDevelopment commented 2 years ago

I am unable to preview a full-image upscale. When upscaling a 1024x1024 PNG using the "Refresh preview (full image)" button, I get a generic error message after about 2 minutes that an error occurred. Looking at the log, it seems like Cupscale or the python process has a time out of around 2 minutes or 100 seconds, after which it will throw an error.

This error does not occur on my desktop PC (Nvidia GTX 1080Ti), it only happens on my laptop (Nvidia GTX 950M). I suspect that the mobile GPU of my laptop is so slow that Cupscale simply declares the upscaling to have failed when I can see that the CUDA cores of my GPU are still doing work (in TaskManager).

Is there any way to increase the timeout?

Full log ``` Running in portable mode. Data folder: C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData Arg: C:\DS3TexUp\Cupscale 1.39.0f1\Cupscale.exe Detected Card: NVIDIA GeForce GTX 950M / GPU: GM107-A / Arch: Maxwell Initialized Nvidia API. GPU: NVIDIA GeForce GTX 950M [DepCheck] CMD: /C python -V [DepCheck] System Python Check Output: Python 3.9.9 [DepCheck] Sys Python Ver: Python 3.9.9 [Servers] Ping to nmkd-hz.de: 37 ms Parsing Patrons from CSV... Found 14 Gold Patrons, 406 Silver Patrons [Servers] Ping to nmkd-cb.de: 200 ms [Servers] Closest Server: Germany (Nürnberg) (37 ms) [MainUI] Dropped 1 file(s), first = C:\DS3TexUp\upscale\m31\a\m31_00_woodplank_10_a.png [ImgProc] Converting C:\DS3TexUp\upscale\m31\a\m31_00_woodplank_10_a.png to PngRaw, DelSrc: False, Fill: True, Ext: UseNew [ImgProc] Color depth of m31_00_woodplank_10_a.png is 32. [ImgProc] Written image to C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\loaded-img\temp.png [ImgProc] Preprocessing C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview\preview.png - Fill Alpha: True [ImgProc] Color depth of preview.png is 24. [ImgProc] Done pre-processing images [CMD] /C cd /D "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch" & python upscale.py --input "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview" --output "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\preview-out" --device_id 0 --fp16 --alpha_mode 0 "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth;50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth;50" [Python] Traceback (most recent call last): [Python] Model: 4x_UniversalUpscalerV2-Neutral_115000_swaG [Python] Upscaling... [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 31, in auto_split_upscale [Python] 1 preview.png.png [Python] result = upscale_function(lr_img) [Python] Image loaded [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 282, in upscale [Python] Reloading model - model_path: C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth@50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth@50 - last_model: None [Python] Setting last_model to C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x-UltraSharp.pth@50&C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\models\ESRGAN\4x_UniversalUpscalerV2-Neutral_115000_swaG.pth@50 [Python] Upscaling image... [Python] output = process(img1) [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 126, in process [Python] output = model(img_LR).data.squeeze( [Python] RuntimeError: CUDA error: the launch timed out and was terminated [Python] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. [Python] For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [Python] During handling of the above exception, another exception occurred: [Python] Traceback (most recent call last): [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\upscale.py", line 375, in [Python] rlt, depth, _ = ops.auto_split_upscale(img, upscale, last_scale) [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 55, in auto_split_upscale [Python] top_left_rlt, depth, current_tile_num = auto_split_upscale(top_left, upscale_function, scale=scale, overlap=overlap, current_depth=current_depth+1, current_tile_num=current_tile_num) [Python] File "C:\DS3TexUp\Cupscale 1.39.0f1\CupscaleData\bin\esrgan-pytorch\utils\dataops.py", line 43, in auto_split_upscale [Python] raise RuntimeError(e) [Python] RuntimeError: CUDA error: the launch timed out and was terminated [Python] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. [Python] For debugging consider passing CUDA_LAUNCH_BLOCKING=1. [ESRGAN] Upscaling Error: Der Index war außerhalb des Arraybereichs. bei Cupscale.Main.Upscale.d__8.MoveNext() ```
n00mkrad commented 2 years ago

There is no such timeout, I'd say the resolution is just too high for that GPU

RunDevelopment commented 2 years ago

I can somewhat confirm that guess. I noticed that 512x512 images did not work, but 400x400 images did.

However, isn't Cupscale supposed to automatically tile large images? It had no problem with upscaling large images (4096x4096) using tiles on my desktop PC. (Cupscale does tile 1024x1024 images but it seems to not tile 512x512 images on my laptop.)

twisted-nematic57 commented 2 years ago

I guess it depends on your available RAM/VRAM....?

RunDevelopment commented 2 years ago

My GTX 950M has 4GB of VRAM, Cupscale uses about 2.5GB of that. My system has about 4GB RAM left available when running Cupscale, and it uses about 1.7GB of that.