Resizable BAR / SAM tanking CPU performance on VKD3D #1406

Open Faugus opened 1 year ago

RX 6800 XT / 5600X

I've been doing some tests and I got the conclusion that Resizable BAR / SAM hurts the CPU perfomance when running games on VKD3D.

This problem doesn't happen on DXVK or Vulkan Native. Also this issue doesn't happen on Windows.

VKD3D games I tested: World of Warcraft, Uncharted 4, Spider-Man, Spider-Man Miles Morales, Resident Evil 2 Remake, Resident Evil Village...

I posted on Mesa's GitLab some tests and comparissons: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7993#note_1702565

I can confirm the findings in Uncharted 4 on an Ryzen 3700x + RX7900XTX.

While SAM is enabled the game is weirdly CPU bound, weirdly because all cores show 40% usage with no core exceeding that. Disabling SAM (in the BIOS) leads to a much more healthy per core usage of 80% and a framerate uplift of +56% on my system.

Tested on an ASUS B450F, 32GB RAM, vkd3d 2.8. No OC or similar.

Copied my results (Ryzen 3700x + RX7900XTX, vkd3d-proton 2.8) from mesa Gitlab:

3440x1440 Ultra	Uncharted 4
SAM ON + `VKD3D_CONFIG=no_upload_hvv`	78 FPS
SAM ON	50 FPS
SAM OFF	78 FPS

I used the heaviest location I could fine for the test (most of the time FPS are higher, but gap stays about the same)

Turning on/off Above 4G decoding and Resizable BAR individually did not show a difference so I merged them into "SAM ON". Reported BAR is Detected VRAM RAM=24560M, BAR=32768M for SAM ON, Detected VRAM RAM=24560M, BAR=256M for SAM OFF.

no_upload_hvv could become a default config for Uncharted 4 until the issue is resolved. Though we have no Nvidia results for now.

Also affects AoE 4 https://github.com/HansKristian-Work/vkd3d-proton/issues/1323, Death Stranding https://github.com/HansKristian-Work/vkd3d-proton/issues/864, and FFVII Remake (only during certain attacks during the Airbuster Boss fight it drops to 20 fps without no_upload_hvv)

Just tried VKD3D_CONFIG=no_upload_hvv with Uncharted and it fixed the issue!

I got a small FPS hit, but now SAM is working properly.

Uncharted	Menu	Ingame
SAM ON + `VKD3D_CONFIG=no_upload_hvv`	141 FPS	138 FPS
SAM ON	146 FPS	88 FPS
SAM OFF	132 FPS	120 FPS

but now SAM is working properly.

From my understanding using no_upload_hvv you're keeping vkd3d from using any BAR memory. So it's not actually "working", but you're avoiding this title-specific performance hit.

I'm really surprised you're seeing a benefit comparing ON + no_upload_hvv vs. OFF. I was not able to reproduce that.

Do you have Resizable BAR option ON in BIOS?

If you do, try to disable it.

Leave only Above 4G decoding ON.

Did a quick re-test. Tested on an ASUS B450F with this months latest BIOS (v5003).

3440x1440 Ultra	Uncharted 4
`Above 4G decoding` + `Resizeable BAR` + `VKD3D_CONFIG=no_upload_hvv`	78 FPS
`Above 4G decoding` + `Resizeable BAR`	50 FPS
`Above 4G decoding` + `VKD3D_CONFIG=no_upload_hvv`	78 FPS
`Above 4G decoding`	51 FPS
everything OFF	78 FPS

Maybe you're not seeing difference because you're running at 4K.

I'm running at 1440p.

Here's a comparison running the game with ReBAR OFF, ReBAR ON and ReBAR ON + VKD3D_CONFIG=no_upload_hvv https://www.youtube.com/watch?v=78XakTs-3bI

VKD3D_CONFIG=no_upload_hvv - Blocks any attempt to use host-visible VRAM (large/resizable BAR) for the UPLOAD heap. May free up vital VRAM in certain critical situations, at cost of lower GPU performance. A fraction of VRAM is reserved for resizable BAR allocations either way, so it should not be a real issue even on lower VRAM cards. https://github.com/HansKristian-Work/vkd3d-proton

I too am seeing this in Uncharted 4, fps in main menu explode from 90fps CPU bound to 160fps GPU bound. Doesn't happen on Windows with native D3D12, or at least it's fast enough to be always GPU bound there.

Dragon's Dogma 2 also suffer from this.

with AMD CPU 3700x on main menu ~40fps without VKD3D_CONFIG=no_upload_hvv
~60fps with VKD3D_CONFIG=no_upload_hvv

Nvidia RTX 407 0- Control suffer from this too. Having ReBar enabled limits the GPU usage to ~65% in Control, depending on location. Disabling ReBar makes the GPU use all power (99%) and adds +15FPS in Control.

What's weird is that VKD3D_CONFIG=no_upload_hvv doesn't seem to do anything for this game, I have to disable Rebar in Bios to actually see a difference. Having rebar off in BIOS is not an option since it seem to hurt performance in other games.

In Horizon Forbidden West it's the opposite.

3440x1440, Very High Preset, 5800x3D+7900XTX VKD3D_CONFIG not set: 105FPS VKD3D_CONFIG=no_upload_hvv: 94FPS

If no general solution can be found it might make sense to create game-specific no_upload_hvv defaults in device.c where it helps performance in most cases.

Yes, I agree. However there might be different results depending on vendor and/or driver. The odd bird here is Control. I hope they figure that one out.

One thing I noticed when having VKD3D_CONFIG=no_upload_hvv set is that a gameplay recording becomes jittery/stuttery.

I can confirm this bug, without VKD3D_CONFIG=no_upload_hvv DD2 and some other games are unplayable.

Operating System: CachyOS Linux KDE Plasma Version: 6.1.3 KDE Frameworks Version: 6.4.0 Qt Version: 6.7.2 Kernel Version: 6.10.1-3-cachyos (64-bit) Graphics Platform: X11 Processors: 32 × Intel® Core™ i9-14900HX Memory: 31.1 Gio of RAM Graphics Processor: NVIDIA GeForce RTX 4090 Laptop GPU/PCIe/SSE2 Manufacturer: Alienware Product Name: Alienware m18 R2

look like Warhammer 40,000: Space Marine 2 also affected by this.

Have ~20% fps improve with no_upload_hvv when cpu bound.

I just tested Warhammer 40,000: Space Marine 2 with no_upload_hvv and it massively REDUCED my FPS (from 100 to 88). I am on 6900 xt/7900X3D/kernel 6.8.12/vulkaninfo --summary says Mesa 24.3.0-devel but inxi -F says Mesa 24.1.3 lol/Proton-Experimental

EDIT: This was a GPU bound scene. on CPU bound scene no_upload_hvv INCREASED my FPS.

@mhmarf Is the game CPU bound?
no_upload_hvv can help with CPU performance but wrost GPU performance.

Indeed you are correct. I tried hard to make it CPU bound but no matter where I looked, GPU was always 98%... therefore I lowered resolution to 800x600 and yes, indeed it reduced my FPS...

800x600: no_upload_hvv -> 117 FPS normal (without no_upload_hvv) -> 100 FPS

2560x1440: no_upload_hvv -> 88 FPS normal (without no_upload_hvv) -> 100 FPS

HansKristian-Work / vkd3d-proton

Resizable BAR / SAM tanking CPU performance on VKD3D #1406