Heavy memory leak. (Eats up 24 GB V-RAM in seconds)

[Sharrnah]

i noticed a very extreme memory leak since some update. I am not sure if its because of the SteamVR Beta or from DesktopPlus, or some strange combination.

I haven't found any issue that mentions this,

As you can see, the DesktopPlus.exe takes around 22GB of Video-RAM already, And it increases every second very quickly until the PC can't handle it anymore and crashes.

Only help is to restart DesktopPlus in the Settings of DesktopPlus and don't touch the Tab in SteamVR.

This always happens as soon as i switch to the DesktopPlus Option in SteamVR. So my guess is that either some Windows Desktop view fills up this memory, or some DesktopPlus Browser window.

I think it happened together with the new SteamVR UI, but could be coincidence.

I am running the current SteamVR Beta (But it happened in some previous SteamVR beta as well.) And the current Desktop+ newui-browser Beta (I think v2.9.9.16)

[elvissteinjr]

NewUI branch builds write log files to the install directory (DesktopPlus.log & DesktopPlusUI.log). Can you post the content of those?

I don't doubt this happens, especially since you mention crashes. However, I'd be careful with trusting the numbers from Task Manager as it used to not decrease the number in this column when applications freed resources. I can see this behavior on my system running Windows 10 as well, but I have no idea if this was fixed for Windows 11 which you seem to be using (article may just not have been updated to mention it). Something like Process Explorer shows a reasonable VRAM number on the Desktop+ process in comparison... at least on my end (250 - 300 MB, 3200x1080 total desktop).

But that's not to dismiss your issue. Was just surprised how I could make Task Manager display higher numbers there than my total VRAM use in the Performance tab. Assuming normal operation, the VRAM use would be not super low, but fairly static. Desktop Duplication may allocate extra textures during a frame update, but they are also freed again right after. That's the happy path at least. You don't have any other overlay setups with different capture methods or special settings, right? Just to make sure I'm checking the same thing.

Hope we can figure something out.

[Sharrnah]

Hi. I looked into the logs and there is nothing out of the ordinary, except maybe one repeated line:

2023-10-25 23:29:18.899              DesktopPlusUI.cpp:144   INFO| Loaded Dear ImGui
2023-10-25 23:29:18.915              DesktopPlusUI.cpp:175   INFO| Finished startup
2023-10-25 23:36:48.595                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-25 23:37:54.690                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-25 23:48:33.487                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-25 23:51:13.777                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-25 23:52:38.619                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-25 23:53:51.377                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 00:03:34.835                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 00:09:01.198                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 00:39:08.509                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 00:40:12.535                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 01:11:46.043                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 01:13:12.209                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 01:13:41.899                  UIManager.cpp:1077  INFO| Restarting dashboard app...
2023-10-26 01:38:35.239              DesktopPlusUI.cpp:527   INFO| Shutting down...

But could be that this are the restarts i did to prevent my VRAM to fill up completely.

Yes i am using Windows 11, but i am using Desktop+ for some time now and never had this before. Memory Consumption was always in acceptable levels. I know that because i often looked at my VRAM consumption because of an application i am working on.

I am not aware of any other overlay setups and not 100% sure what you mean. Do you mean other overlay applications that i might have running?

When my VRAM fills up, the PC becomes unresponsive until it eventually locks up. (every action in windows or in SteamVR takes forever) because of no more free VRAM. It might not hard crash immediately, but you notice everything slowing down more and more.

I uninstalled the SteamVR Beta and see if that does help. And i will try to reset the Desktop+ overlays etc to see if maybe some overlay is causing issues.

I will keep this updated when i find anything.

[elvissteinjr]

I was hoping to see the full logs, especially from DesktopPlus.log. System details and desktop layout logged at the beginning usually help me with replicating the scenario on my end.

The logs about restarting dashboard app should indeed be from your manual restarts. This is only triggered from the troubleshooting section and when resetting all settings.

The SteamVR Beta is now stable, so there shouldn't be much difference. Unless you were to roll back manually, but ultimately it won't help us as this shouldn't happen in the first place.

Regarding overlay setups I just meant how you possibly have multiple overlays with various settings configured in Desktop+ itself. For example, a window capture would point me to look at the Graphics Capture code instead. Actually if you don't mind, having your config file (config_newui.ini in this case) would be helpful as well for troubleshooting.

[elvissteinjr]

I've tried to do some digging on my end. And while I seemingly found some kind of leak when resetting the capture (happens on display change events or when toggling Single Desktop Mirroring), the VRAM use normalizes after about a minute. This might be belated cleanup on SteamVR's end since it doesn't happen if I never sending the texture to it, but easy explicit texture cleanup functions don't change the VRAM situation. I need more time looking into that specifically. Full restart works of course, but is infeasible once stateful browser overlays come into play.

But even that case doesn't explain yours, as here nothing fills up in seconds or on its own (spamming the setting to force a reset can get it far, but this isn't that).

As said I'd love to see the full logs, even if you think there's nothing useful in there.

[Sharrnah]

Hi and sorry for the delay. I Finally found the culprit.

If i set it to use "Desktop Duplication" for the desktop overlay, it results in this heavy memory leak.

If i use "Graphics Capture" everything is fine.

[elvissteinjr]

Right. If it does work for you we may leave it at this. Asking around I couldn't find anyone else able to replicate this, which makes it troublesome to fix. This may be some isolated case, maybe some external factors ruining things... or maybe it's just my fault, I honestly don't know.

The Desktop Duplication code is planned to be rewritten at some point (current version is adapted from Microsoft sample code, hopefully correct stuff), but it's far down the todo list. Though that wouldn't necessarily guarantee a difference in memory behavior.

If you do want to continue troubleshooting, all I can is ask for full log files. SteamVR's logs may help as well (stored in "Steam\logs").

[Sharrnah]

DesktopPlusUI.log DesktopPlus.log

here are the full log files.

Another thing i just remembered. I had an issue for quite a while where my second monitor did not work as overlay but produced a very very very wide overlay screen with no image. (It was insanely wide i could not even see its end, and i am pretty sure i don't have such kind of a widescreen monitor).

Could not reproduce that now, so not sure if related and not sure what setting i had there.

[elvissteinjr]

2023-11-29 23:13:02.558 OutputManager.cpp:388 INFO| Using cross-GPU copy

This is the only thing that jumps out to me, though it's only on the last execution so it might be nothing. Cross-GPU copy is used when the mirrored desktop is on a different GPU than the one SteamVR tells the application to use. Under normal circumstances ("Single Desktop Mirroring" was not turned on manually), Desktop+ mirrors all displays from the GPU the primary desktop is on. Mirroring desktops across multiple GPUs isn't supported via Desktop Duplication at this time, but should only result in those staying blank.

The kicker here is that the logged display listing says that the primary desktop is on the 3090 in all cases, so that is odd. But as it only happened in one instance it seems like it's not the cause of this issue at least.

Speaking of the desktops however, I see that you got one display connected to the 3090 and one to your iGPU. Any particular reason? The Varjo itself appears to require one DisplayPort, so that would leave some free ports on the GPU. If you can't plug that display into the 3090, can you at least try with it disconnected to see if it makes any difference? This is best I can think of for now after looking at these logs.

I had an issue for quite a while where my second monitor did not work as overlay but produced a very very very wide overlay screen with no image. (It was insanely wide i could not even see its end, and i am pretty sure i don't have such kind of a widescreen monitor).

Insanely tall overlays can happen with weird cropping values or wrong image dimensions being reported somewhere in the chain, but very wide ones sound weird, unless the width property of the overlay got corrupted somehow (SteamVR overlay sizes are defined by their width). Hard to say much without being able to reproduce this.

[Kegetys]

Looks like this issue is caused by the vr::VROverlay() "GetOverlayTexture" and "ReleaseNativeOverlayHandle" pairs. Some GPU resource is maybe not getting released properly there. The use appears correct as far as its possible to tell from the OpenVR "documentation" so maybe it is a OpenVR/SteamVR bug.

I was able to work around the issue by making GetOverlayTexture get called only once, and storing & re-using the ID3D11ShaderResourceView it returns. Not sure if it would need to be refreshed in some places though, at least my single desktop overlay works fine with a hacky fix I did. It seems to have significantly improved stuttering I was getting when running low on VRAM (ie. playing DCS World in VR)... My guess is some GPU surface is created but never actually used so it doesn't use "real" VRAM, but causes problems with the WDDM memory management.

[elvissteinjr]

This is very interesting, thanks for looking into this.

I suppose the easy way out here would be to disable partial desktop texture updates and always do a one with SetOverlayTexture(). I'm not sure to what extent keeping the shader resource view around is safe. The current code is written this way since I assumed it would lock until I'm done with it, preventing changes to show up, but this isn't 100% clear. I could see it break if SetOverlayTexture() were to create a new backing texture (i.e. on size changes), if nothing else.

Also knowing this context, what might contribute to this issue is how Desktop+ feeds back a texture owned by SteamVR as a shared handle for Desktop Duplication overlays. This saves performance and VRAM for additional overlays, but don't ask me if it's actually allowed. Hasn't been an issue for over 3 years, though. I know that GetOverlayTextureSize() leaked in a similar way when I tried to use that. Use of that function is avoided throughout the application because of this, but it might be related to that if I think about it. I haven't sat down and checked it, however.

I'll need to see what the best way to handle this. On a 3090+ grade GPU it's probably no issue to not have partial updates. Perhaps I can get clarification on the GetOverlayTexture() behavior from Valve. If the handles can be kept around it'd be the way to go.

Can you provide some details on your system? GPU + driver version perhaps? I definitely can't reproduce this on my end. Even when desktop duplication hits 1000+ fps (via high update rate mouse cursor movement) there's nothing worth reporting, but I'm also sitting on a 1070 still. The issue itself doesn't seem to be too widespread either since I haven't heard anything about it outside of here yet.

[Sharrnah]

still following this closely and thanks for looking into this.

In the meantime i think i noticed this happening also in the other Desktop overlay mode. It just seems that it randomly starts happening.

Sometimes i play for 1-2 hours and when i ocassionally check the memory usage, its fine. But then suddenly the memory reported in task-manager skyrockets.

SteamVR Lagging and crashing might be related but also not 100% sure.

[Kegetys]

I'm running: Windows 10 GeForce 2080ti with driver 546.65 Latest non-beta SteamVR (Should be 2.2?) Vive Pro I have a single desktop duplication overlay in Desktop+, which is rather large since it covers a 4k monitor + 1920x1200 in portrait. Should be 5040x2160 total, both monitors and the vive are plugged to the 2080ti. Nothing unusual from my setup that could be relevant comes to mind.

I see the dedicated gpu memory number in task manager for DesktopPlus rise rapidly to tens and then hundreds of gigabytes when the overlay is visible, but not the graph under 'performance' tab. Everything appears to work ok for me even when the numbers reach terabytes for DesktopPlus, except when something like DCS uses up all the "real" VRAM (performance tab graph is pegged at close to maximum). Then I start to get extreme stutters where the game and SteamVR compositor are freezing for 1-30 second at a time. Without DesktopPlus running or with my hacky fix, I get only very small hickups.

[Sharrnah]

seems to also happen with browser overlays (using the newUI Preview + Browser Beta) and not only desktop overlays.

[elvissteinjr]

All overlay update methods across the application do use GetOverlayTexture() in some capacity, so that does sound like it would happen with our current theory. Graphics Capture overlays only use it to share textures between multiple overlays with the same capture target, so this should be the least affected overlay type. Same goes for Performance Monitor and the VR UI. For the those it's usually only called once on creation, though.

I haven't had the time to work on a solution yet. Was hoping to hear back from Valve, but nothing so far. This has some wider-ranging implications for the texture sharing part, so I don't want to just rush into it.

[elvissteinjr]

Sorry for the delays. I did hear back from Valve and got the okay for keeping the handle around as long as I want (doesn't seem to be any locking mechanism but just passing the SRV along with AddRef/Release() pairs). They weren't able to tell me why this would leak on your systems though, so not any smarter there.

With that confirmation I've done some work on reducing shared texture allocations to the minimum necessary for it to work. While this won't fix the issue you're experiencing, it should greatly mitigate it. Re-allocations now only happen on resize or other resets. For Desktop Duplication this is only when the resolution changes or display gets reset (i.e. exclusive fullscreen game). For Graphics Capture this is on any resize, so it is more likely to encounter. Still, a far cry from having it happen every frame update. Normal VRAM use is about the same as before, however. Kind of to be expected on systems where immediately releasing the shared texture did actually free the memory of it.

I haven't adapted Desktop+ Browser to this yet, but otherwise it's good to go (NewUI branch only). Please try it when you have the time. Tell me if you aren't able to build from source yourself (Kagetys obviously can, but not sure about Sharrnah). I'll provide a build then (regular next build isn't quite ready yet).

Another thing with this that shouldn't happen, but is worth looking out for, is overlays stopping to update. Minimizing shared texture allocation also meant putting assumptions in place how long the backing texture is valid. I do believe this is mostly ironed out in the current code, but please report if you see anything stop working.

Thanks!

[Sharrnah]

Thanks. i was able to build it myself.

I will report back if i find anything or not.

[LadPack]

Also running into this issue, but don't have the skills to build myself. Would it be possible to get a built version of the above?

I ended up uninstalling it as it was causing some frustration while trying to move overlays in sim racing

[elvissteinjr]

Sure, here's a build. Situations like this are asking to finally have some CI/CD to set up, but this will do for now. DesktopPlus-NewUI-SharedTextureTest.zip

[Sharrnah]

i was able to try it yesterday. (hadn't got into VR until now).

Seems to have worked but i did not try it for very long so i would need more time to confirm.

[LadPack]

Spent about an hour in iracing today, overlays worked well. New UI worked well. Didn't see memory leak. I'll pull up the performance monitor next time I am playing to double check, seems to be fixed

[Kegetys]

I tested the new build also and the problem appears to indeed be fixed and at least my single desktop overlay still works well.

I got a new GPU in the meanwhile too (4080 Super) and with it I am still able to repro the issue in the old version.

[LadPack]

This may be the wrong place for it. But since installing the build I've had some issues getting steam vr to launch. I uninstalled the old desktop+ and installed the new one in program files. When I try to launch steamvr I get a "not initialized 109" error as well as another error about not launching a background app.

I don't think I've changed anything other than desktop+. Any idea what could be happening?

I can eventually get it to launch through a combination of launching desktop+ itself and starting/stopping steam vr. But I'm not sure the exact sequence to get it going again.

Desktop + is enabled as an app on the steam vr settings

On Sat, Mar 9, 2024, 09:25 Keijo Ruotsalainen @.***> wrote:

I tested the new build also and the problem appears to indeed be fixed and at least my single desktop overlay still works well.

I got a new GPU in the meanwhile too (4080 Super) and with it I am still able to repro the issue in the old version.

— Reply to this email directly, view it on GitHub https://github.com/elvissteinjr/DesktopPlus/issues/64#issuecomment-1986870782, or unsubscribe https://github.com/notifications/unsubscribe-auth/A43SXOSMWWN3ZNCCPYI5Z3DYXMLVHAVCNFSM6AAAAAA6QKIWYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHA3TANZYGI . You are receiving this because you commented.Message ID: @.***>

[elvissteinjr]

Having both the Steam version and non-Steam version around can lead to some conflicts. Both are the same binaries (in case of same version), but Steam manages the app entry for the SteamVR applications list when the Steam version is installed.

Haven't seen the issue you describe in this context yet, though. What I have seen is input bindings not loading or be stuck with whatever is installed by Steam as its manifest takes priority.

Application manifests are usually removed automatically from the list when the target files don't exist while SteamVR is launched. But you can also do this manually by editing "Steam\config\appconfig.json". Removing the non-Steam build's manifests should get rid of Desktop+ getting involved at launch.

This is just in general though. If this doesn't help we'd have to dig deeper. SteamVR has log files in "Steam\logs". Desktop+ has some too, but I don't expect them to have more than that error message in it if it failed to init in the first place.

[Sharrnah]

i think this is fixed. :) Never had any issues again. (sorry. forgot about this)