search

DeclanRussell / NvidiaAIDenoiser

A simple implementation of Nvidia's AI denoiser
MIT License
398 stars 52 forks source link

High pixel count "[OptiX]: Unknown exception" - Possible VRAM limitations #8

Closed jackjt8 closed 5 years ago

jackjt8 commented 5 years ago

Noticed some odd behaviour when working with high resolution / pixel count PNG images; Mainly seeing [OptiX]: Unknown exception. Initial assumption is that we are running out of VRAM but it's a bit more confusing than that once you look into it.

GPU: GTX 1060 3GB 417.01 OS: Win10 Home v1803 b17134.407

Resolution Filesize(MB) VRAM(GB)
3840x2160 15.9 1.8
5333x3000 28.6 2.3
6222x3500 37.6 2.4
7111x4000 47.3 2.3
7340x4129 49.9 [OptiX]: Unknown exception
7680x4320 84.0 [OptiX]: Unknown exception

Two things stand out here. The first is the VRAM usage does not seem to exceed 2.5GB. The second is that there appears to be a limit of around 30mp for the source PNG image.

Someone with a GTX 980 Ti 6GB has also noted issues with high resolution / pixel PNG images where only 5GB of 6GB of VRAM can be used, and the limit is around 50mp for the source PNG image. Reddit comment source

Now, while I already know that OptiX 4.0+ does not fully support Out-of-core features (hence we can only use VRAM and not shared memory), what I don't know is why we are not able to fully use the VRAM these cards have available. I'm not sure where the issues lies but I would appreciate any insight in to what is going on.

DeclanRussell commented 5 years ago

So OptiX allocates a bunch of memory on its own on top of the space that the app allocates for the input textures. This memory is relative to the properties of your card and what you're doing with OptiX. I'll need to dig deeper into this to know for sure but at a glance I think this is the problem. Its still a little suspicious to me that you're having issues with 7340x4129 size images and beyond as this should uncompressed onto the GPU to around 500mb which is still way below the total memory available on your card.

The good news is that you can limit the memory that the denoiser uses. I've been meaning to add support for this for a while now but in the version of OptiX that the current release uses its a little flaky. I was hoping to wait until the next version of OptiX comes out but if this is blocking you I can look into creating a new release with support for the feature which may help.

The only work around I can think of for the mean time would be to divide these images into smaller images, denoise them separately, and then reconstruct the image. Far from ideal I know but in theory it should work fine.

jackjt8 commented 5 years ago

I had figured that would have been the case, but I wasn't certain given my limited experience. It would make sense given how this allocation scales between a 1060 3GB and a 980 Ti 6GB (from .5 to 1GB).

I'll try and run a few more tests with some lower resolutions and a few different formats to see how the memory is affected. I honestly don't know why the memory usage is so high. You don't have to create a new release- I'm just experimenting with this for a hobby of mine. While it is restrictive, I'm not too concerned about it as I have other methods for denoising which, while not as impressive as the AI, they don't product artifacts. The tiling method should work around the issue just fine and I'll use it in the mean time when needed.

I'll provide one of the problem renders just in case you would like it for testing purposes Google Drive downloadlink 7680x4320.

DeclanRussell commented 5 years ago

I've tested this with the latest release and after testing your provided image is seems that this has been fixed by the upgrade to OptiX 6. Let me know if you continue to have any problems.