[D3D11] Outward VRAM usage keep increasing every time loading new location

[HenryJk]

VRAM usage keep increasing every time loading new location. Eventually it leaks to RAM and degrade performance, usually loading a new location a single time is enough to make it leaks to normal RAM. Suspected objects not being unloaded properly.

Software information

Outward, medium preset with advanced shader turned on.

System information

• GPU: GTX 1650 Max-Q
• Driver: nvidia 460.56-7
• Wine version: wine-staging v6.1
• DXVK version: v1.7.3-37-g3f91cdbc

Apitrace file(s)

• Outward.trace

Log files

• d3d9.log:
• d3d11.log: Outward_d3d11.log
• dxgi.log: Outward_dxgi.log

Going out and in of a location increase VRAM usage and RAM usage.

[doitsujin]

DXVK does not free VRAM allocations by design, this is no memory leak. If there is one, it's almost certanily the game's fault. DXVK_HUD=memory shows the actual memory usage of the game.

Now in theory, DXVK could evict unused resources to system memory and move resources that are actually needed into VRAM to work around resulting performance issues, but that would require a large-scale rewrite to work well, which is why that hasn't happened yet and probably never will.

Also, the apitrace doesn't really contain anything useful since it just ends after a handful of black frames.

[HenryJk]

If the resource evicted from VRAM is actually unused, why would that cause any performance problem? Also, does apitrace actually work with steam game? I have tried multiple times and apitrace doesn't behave as I expect.

[Kron4ek]

I can't reproduce this on AMD Polaris (though with DXVK 1.8.1-1c2edab) with Mesa 21.0.0, changed location about ten times and VRAM usage was still about the same, so this is probably Nvidia-specific. Does this apply to all locations or only the ones from the screenshots?

[HenryJk]

VRAM usage is via mangohud, and somehow, dxvk-hud doesn't report such increase. It applies to all locations but I do use nvidia. Aside from VRAM usage, the fps is also degrading everytime loading a new location.

[Kron4ek]

Ok, now i tried with MangoHud and can confirm that VRAM usage value is different in DXVK and MangoHud. According to MangoHud, VRAM usage is stuck at 3.7 GiB (and i have 4 GB, so this is max) and RAM usage increases a little with every loading of location, so it looks like memory leak indeed, if the values are not wrong.

I also tried to run the game with WineD3D and there is no memory leak in this case (again, according to MangoHud), so seems like a DXVK or Vulkan-driver bug.

[HenryJk]

Do you get fps degradation as well? I won't mind memory increase if they aren't actually used, meaning don't degrade performance. The thing that bothers me is the fps degradation every loading screen. The degradation is especially huge on a town. If I load the game when my character already in town, I can maintain 60 fps. But when going out and in of a town, the fps start to tank.

[Kron4ek]

No, to my surprise, i don't get FPS degradation.

[HenryJk]

The worst degradation is between Abrassar Desert and Levant, or Abrassar Desert and Enmerkar Forest. For the latter, my FPS can tank to the 30s.

[K0bin]

If the resource evicted from VRAM is actually unused, why would that cause any performance problem?

DXVK will create new resources in system memory when it runs out of VRAM. What doitsujin meant was that DXVK will never move resources around.

And yes, the MangoHUD metric is not relevant here. Focus on the DXVK HUD one. There probably isn't much we can do here so just lower your texture resolution.

Either way, can't do anything without an apitrace.

[Kron4ek]

apitrace (4.6 GB when extracted).

[HenryJk]

How to create apitrace, really? Tried to do the command apitrace-latest\bin\apitrace.exe trace -a dxgi Outward.exe on windows and it seems to not working

[Kron4ek]

I used exactly the same command (on Wine), works for me.

wine apitrace-latest-win64/bin/apitrace.exe trace -a dxgi Outward.exe

[HenryJk]

New apitrace, this time it works properly Outward.7z

Changing between these 2 locations have a performance hit. Also use dxvk release build instead of the one from proton-ge.

Also tested with wined3d and the performance doesn't degrade after every loading, although the fps is lower to begin with. With wined3d, the fps in first and third scene is 68-70 and the second scene is 20-25. The performance was not great but stable.

[doitsujin]

wined3d doesn't have issues with that because the OpenGL driver does all the black magic for it. DXVK on AMD cards should also be more or less fine since the driver itself has more dynamic memory management than Nvidia's does.

This kind of thing usually happens when the game decides to recreate its render targets and those then end up in system memory, but as mentioned, there's pretty much nothing we can do about that without rewriting a lot of stuff. Lowering texture quality in game might help mitigate the issue, or changing the amount of VRAM reported to the game via dxvk.conf.

Will look at the trace tomorrow, but I don't really consider this a bug. There's some hacks we could attempt (like limiting the amount of memory used for textures vs render targets on Nvidia GPUs), but doing that will regress other games.

[HenryJk]

Correct me if I'm wrong, but basically the only way to solve this issue is by writing garbage collection for gpu? I wonder why is the unused garbage still affecting performance.

[TheZoq2]

This issue seems to affect DCS world and squad, so I'd really appreciate a fix though I fully understand it being a massive amount of work.

DXVK on AMD cards should also be more or less fine since the driver itself has more dynamic memory management than Nvidia's does.

I'm seeing these issues in DCS world on both my old rx 580 and new rx 6800. (game runs fine, then suddenly drops down to sub 5 FPS, especially in texture-heavy environments. IIRC, this doesn't happen right after VRAM usage goes above 100%, I've seen 150% etc. without issue. Maybe that's the render target thing you talked about.

Correct me if I'm wrong, but basically the only way to solve this issue is by writing garbage collection for gpu? I wonder why is the unused garbage still affecting performance.

As I understand it, because it's never moved out of the GPU. When new textures etc. have to be loaded, those have to go in system RAM instead of VRAM.

[doitsujin]

I don't really know what to do here since the only real solution is to rewrite half of DXVK to work around these games being shit which is obviously not really viable.

[TheZoq2]

Yea, it's an unfortunate situation. In the DCS case, it seems like lowering texture quality a bit helps, and the problems are only happening on one map for me after my GPU upgrade.

[Bxaa]

Same issues in dx9 (all gamebryo games TES-4, F3\NV) vram overflow and slide show 1-5 fps. After loading new location vram not purge. Finally dxvk start use shared adapter memory :(

(*) Simple test - just go to interior\exterior location few times (no new textures on level, but used vram start to increase. Seems to be it is a memory leak or global memory manager issue dxvk versions dx9\10\11) Maybe wrong vram size detecting? (dedicated + shared?)

Is there way to cap vram usage for dxvk? d3d9.maxAvailableMemory not affect

@doitsujin, hi! i'm use your build 1.8-21-gd51562fc9 (fixed regression dx9) (Win 10, 6gb vram) My conf: d3d9.maxAvailableMemory = 4096 d3d9.samplerAnisotropy = 16 d3d9.invariantPosition = True d3d9.evictManagedOnUnlock = True d3d9.dpiAware = True d3d9.deferSurfaceCreation = True d3d9.floatEmulation = True d3d9.shaderModel = 3

Notice: (1.8.1 still randomly crash game/tested with Oblivion) Your custom build 1.8-21-gd51562fc9 - no such issues Do you need trace?

[TheZoq2]

Is there way to cap vram usage for dxvk? d3d9.maxAvailableMemory not affect

As I understand it, that just tells the game how much VRAM is available. But as far as the game is concerned, it has unloaded the textures from the previous level, it's just that dxvk does not do that in reality. So .maxAvailableMemory might fix the problem if the game does something like pre-caching a bunch of texture because the space is available.

[misyltoad]

You need to use both memoryTrackTest and the memory amount one to limit the vram in d3d9

[Bxaa]

@Joshua-Ashton Thx, memory track and max memory seems to be fixed memory leak Tested with tes-4 and dxvk build 1.8-21

_Is it for dx9 version only? (memory track)_

_EDIT: _After more testing - still same issue, but with memoryTrackTest it is not intensive__

[Bxaa]

So after test: max available memory - a reason of issue. (in my case) Without it all work fine. Maybe will be useful to someone. (dx9)

[adamreichold]

I am sorry if this unrelated, but I noticed a similar issue with MudRunner running under Proton Experimtal: More detailed maps make the game being killed by the OOM killer after taking up more than 14 GB of memory. (This is on integrated graphics with shared memory, but since the memory is not attributed to the process, I guess that it is graphics memory.)

Using WineD3D, the game loads and runs those maps without issues. What is even more interesting, if I load the map using WineD3D and then resume it (instead of initializing a new game) using DXVK, things work as well. So it seems, the game's initialization routines interact in some really unfortunate way with DXVK's memory management.

EDIT: This radv/amdgpu in versions 22.3.2/6.1.3.