search

Facepunch / sbox-issues

173 stars 11 forks source link

Volumetric fog vram leak #5983

Open bub-bl opened 1 month ago

bub-bl commented 1 month ago

Describe the bug

WHen i launch the editor, the editor crash after few seconds or when i move the scene tab or the game tab

To Reproduce

Expected behavior

The editor will dont crash

Media/Files

The log file: Log.log

Another log file with another error: Log.log

core_4.nv-gpudmp.json

https://github.com/user-attachments/assets/42093930-5c22-4c8d-ac5e-330df42c37d1

Additional context

Windows 11 Nvidia 2060 SUPER (Latest drivers update)

handsomematt commented 1 month ago

Has this started happening recently? Can you run vulkaninfo and provide results https://nvidia.custhelp.com/app/answers/detail/a_id/5429/~/running-vulkan-info

bub-bl commented 1 month ago

Has this started happening recently? Can you run vulkaninfo and provide results https://nvidia.custhelp.com/app/answers/detail/a_id/5429/~/running-vulkan-info

I've had this problem since ~10pm or ~11pm, an update was released, I restarted the editor and since then I still have the problem.

This is the result: infos.txt

handsomematt commented 1 month ago

I don't think we've pushed anything rendering related tonight.. It's been fine yesterday and before?

Only thing I can ask you to try now is uninstalling Overwolf as it fucks with a lot of shit.

bub-bl commented 1 month ago

I still have the same problem after completely uninstalling Overwolf + Restarting the computer

I've never had any problems since I've been using the editor, I haven't done anything special, I haven't installed anything on my computer. The problem occurred all of a sudden for no apparent reason.

bub-bl commented 1 month ago

I have created a new project and i dont have this problem with them. I just will disable all my gameobject and re enable it to see whats cause this issue

bub-bl commented 1 month ago

Ok, I think I've found the real problem... I've created a material with a render target that displays the camera on a plane. I'm pretty sure I must be exceeding some rendering limit or something.

Other than that, I don't know what it could come from. https://github.com/user-attachments/assets/0c67c580-9727-440b-bc03-b972365ccaf5

handsomematt commented 1 month ago

Awesome, thanks for the thorough diagnosis. I'll take a look.

bub-bl commented 1 month ago

Awesome, thanks for the thorough diagnosis. I'll take a look.

You should take a look at Camera.RenderToTexture, the problem occurs exactly on this line.

Aditionnal informations:

I have played with the texture size, the problem seem to be produced with low resolution and non power 2

// Crash with the Camera.RenderToTexture: https://github.com/user-attachments/assets/79c80852-e948-40f4-bf92-efcea02b803f

// Another strange spaming message when testing a lot of things:

https://github.com/user-attachments/assets/26ede3ba-a748-4328-8935-0a4f23bf7ef7

The console log messages:

First message:

2024/07/29 13:29:21.4712    [engine/RenderSystem] TEXTURESTREAMING: Extremely low memory: Available mem: 0.00 MB, Required: 2.09 MB.    
2024/07/29 13:29:21.4712    [engine/RenderSystem] rendersystem (Vulkan) tracked resources:  
2024/07/29 13:29:21.4712    [engine/RenderSystem]   VB: 96 buffers, 193.45 MB   
2024/07/29 13:29:21.4712    [engine/RenderSystem]   IB: 57 buffers, 16.04 MB    
2024/07/29 13:29:21.4712    [engine/RenderSystem]   CB: 1363 buffers, 2.59 MB   
2024/07/29 13:29:21.4712    [engine/RenderSystem]   TEX: 2342 textures, 6481.63 MB  
2024/07/29 13:29:21.4712    [engine/RenderSystem]   TOTAL: 6693.71 MB   
2024/07/29 13:29:21.4712    [engine/RenderSystem] Texture Streaming:    
2024/07/29 13:29:21.4712    [engine/RenderSystem]   6,481 / 6,427 MB (non-evictable: 6,348 MB)  
2024/07/29 13:29:21.4712    [engine/RenderSystem] Overall System GPU memory use:    
2024/07/29 13:29:21.4712    [engine/RenderSystem]   7,331 / 7,284 MB

Last message:

2024/07/29 13:31:04.2904    [engine/RenderSystem] TEXTURESTREAMING: Extremely low memory: Available mem: 0.00 MB, Required: 1.95 MB.    
2024/07/29 13:31:04.2904    [engine/RenderSystem] rendersystem (Vulkan) tracked resources:  
2024/07/29 13:31:04.2904    [engine/RenderSystem]   VB: 96 buffers, 193.45 MB   
2024/07/29 13:31:04.2904    [engine/RenderSystem]   IB: 57 buffers, 16.04 MB    
2024/07/29 13:31:04.2904    [engine/RenderSystem]   CB: 1394 buffers, 2.91 MB   
2024/07/29 13:31:04.2904    [engine/RenderSystem]   TEX: 2563 textures, 7651.46 MB  
2024/07/29 13:31:04.2904    [engine/RenderSystem]   TOTAL: 7863.86 MB   
2024/07/29 13:31:04.2904    [engine/RenderSystem] Texture Streaming:    
2024/07/29 13:31:04.2904    [engine/RenderSystem]   7,651 / 6,427 MB (non-evictable: 7,635 MB)  
2024/07/29 13:31:04.2904    [engine/RenderSystem] Overall System GPU memory use:    
2024/07/29 13:31:04.2904    [engine/RenderSystem]   8,482 / 7,217 MB
bub-bl commented 1 month ago

I have set the texture resolution to 512x512. using a power of 2 resolution seem to "decrease" the crash frequency

Also, i have let the game running a long time, the editor crash after ~15 minutes without touching anything.

2024/07/29 14:15:19.8680    [engine/Engine] C:\build\_work\sbox\sbox\src\rendersystem\texturebase.cpp(3519):ResourceHandleToData(  ) failed! Falling back to error texture!
handsomematt commented 1 month ago

Do you think you could do a minimal .sbproj reproduction of this? My time is minimal currently but I'd love to get this fixed for you asap

bub-bl commented 1 month ago

Do you think you could do a minimal .sbproj reproduction of this? My time is minimal currently but I'd love to get this fixed for you asap

Sure i can try to reproduce this, i will add you in a github repo when the its ready. Thank you !

bub-bl commented 1 month ago

I've tried a whole bunch of things in a new project (in an empty scene), the problem doesn't seem to be happening. I think it must be an accumulation of a lot of things. Graphics memory overflow or something like that. In my other project, the problem always occurs when I do something very specific, but doesn't seem to happen in the new project when reproducing the same thing.

For the moment, I've found a solution that greatly reduces the frequency of crashes, checking my entire project and removing all errors linked to missing textures, non-compiled materials etc. This seems to alleviate my problem. I think I'm an isolated case and that the problem must surely occur in a very specific case depending on the hardware configuration used.

handsomematt commented 1 month ago

There was a memory overflow with cubemaps, may have been triggering this one. See if it's made a difference I suppose.

bub-bl commented 1 month ago

The problem is not solved on my side, but it only trigger when i use a resolution with a non power of 2. So for now i use this hack to avoid this issue

handsomematt commented 1 month ago

Can you add me to your git repo and tell me what makes it crash

bub-bl commented 1 month ago

I have finally found the issue !

The problem is caused only when no gpu memory is available to generate the texture. But that is not the real problem. The real problem is related to the wrong gpu memory cleaning.

When you load your scene, some textures are loaded in the GPU. But these textures are not cleaned when the scene is stopped. That cause a gpu memory overflow.

If you run and stop a lot of times your scene, you can see a lot of not unloaded and duplicated textures in GPU Resident Texture window.

When i load my scene for the first time, i have ~300 textures loaded. After some start / stop scene action, i got ~1100 textures loaded in the GPU without doing nothing, just start / stop the scene. And you can see in the video bellow, i have the same duplicated multiple times texture and these textures are not cleaned.

https://github.com/user-attachments/assets/fcfe3c23-36d0-4031-96ed-9ee184b06b3f

handsomematt commented 1 month ago

Awesome analysis! Should be able to fix it from that.