search

doitsujin / dxvk

Vulkan-based implementation of D3D8, 9, 10 and 11 for Linux / Wine
zlib License
13.03k stars 836 forks source link

Escape from Tarkov (DX11 Unity 5 game) crashing on map load when using more than declared RAM #873

Closed Faalagorn closed 5 years ago

Faalagorn commented 5 years ago

That's quite odd issue that I've been trying to troubleshoot for the previous couple of days. Mind me being beginner, so it may be something obvious I'm missing.

Escape from Tarkov runs generally without issues on DXVK, but I am trying to see if it'll run on 4 GB for the friend of mine. The PC I tested it have 16 GB of RAM (2x 8 GB), so I used the mem=4G kernel parameter to limit my RAM, and gave Linux 8 GB swap partition. The game generally runs fine in menu and in the small "Factory" map, but when loading bigger "Customs" map, the game starts to use more than 4 GB of RAM (it's 64-bit executable), swap is slowly being filled when the map load, but the Unity crashes with the attached error when the map finishes loading, timer goes to "0" and actual gameplay tries to load. It only happens with DXVK – the game loads on Windows and Wine's D3D11 using swap, only stutter is more noticable obviously because of the swap being used instead of RAM.

I tried all kind of settings, including setting vm.swappiness=100, zswap.enabled=1 zswap.compressor=zstd zswap.max_pool_percent=100 zswap.zpool=z3fold kernel parameters for zswap, I run the game on separate X server, so there's nothing running in the background except st htop (for monitoring), xorg, systemd and the game itself (running on ALSA), I've read through entirety of #747 and tried all the tweaks, including:

  1. Modify /etc/tmpfiles.d/local.conf to disable Transparent Huge Pages entirely: 
    w /sys/kernel/mm/transparent_hugepage/enabled - - - - never
w /sys/kernel/mm/transparent_hugepage/defrag - - - - never
  2. Fiddling with dxvk.conf, including 
    dxgi.maxDeviceMemory = 1024
dxgi.maxSharedMemory = 1024

    to limit GPUs memory below 4 GB (this PC runs NVIDIA's GeForce GTX 960 with 4 GB VRAM) and/or

    dxvk.allowMemoryOvercommit = True

    to allow using more memory, and various other dxvk.conf variables. I also tried DXVK_STATE_CACHE=0, but it didn't help.

I use the tkg's PKGBUILD with DXVK embedded, running the latest master of everything, and I'm now in the process of building nvidia-all from it to see if it changes anything. I didn't try using custom kernels, but I doubt it will change anything (I'm running stock 4.20.3.arch1-1 kernel from Arch's repos).

I can't test whether it's the mem=4G kernel parameter that's causing issues and whether the game would ran fine with regular 4 GB of RAM, as I don't own a 4 GB DDR3 stick.

It's also worth noting that the game also runs fine on my 16 GB RAM with AMD's Radeon RX480 with 8 GB VRAM rig, but limiting RAM to 4GB hangs my CPU as I don't have swap at all.

Is there anything I can do to make it use swap and prevent it crashing with 4 GB? Will it crash on physical 4 GB too and I am forced to run this particular map on Wine's D3D11 renderer? Thanks in advance for any pointers :). I'll let you know if I find something new too, and will try to get my hand on a working 4 GB RAM stick to hopefully test it proper.

Software information

Escape from Tarkov 0.11.1.2474 (Unity 5.6.6f2, D3D11 renderer), 1024x768, all settings to lowest. The game have "ram cleaning" option, I tried it both on and off. I can also post content of my game settings files if needed.

System information

Apitrace file(s)

Since it's not crashing with regular D3D11 I didn't do apitrace, as from what I undestand I have to do that without DXVK, but I can do so if instructed (I never done it, but guess I'll manage to do so if needed).

Log files

Since the game launches with dedicated launcher, there's also BsgLauncher_d3d11.log, BsgLauncher_dxgi.log, CefSharp.BrowserSubprocess_d3d11.log and CefSharp.BrowserSubprocess_dxgi.log, but the launches closes as soon as the game's launched, so it shouldn't matter, but I can attach them if needed.

I generally launch with export WINEDEBUG=-all especially since when I tried to launch it with reverse export WINEDEBUG=+all the game didn't want to run. Wine generally does not crashed too, but it did once and allowed me to save the backtrace.txt, so I can upload it too.

K0bin commented 5 years ago

FYI

dxgi.maxDeviceMemory = 1024
dxgi.maxSharedMemory = 1024

doesn't actually limit VRAM usage, it just makes DXVK report that your game only has 1GB of VRAM. The game might allocate more anyway.

doitsujin commented 5 years ago

Is there anything I can do to make it use swap and prevent it crashing with 4 GB? Will it crash on physical 4 GB too and I am forced to run this particular map on Wine's D3D11 renderer?

This sounds a lot like issues I've had before on both AMD and Nvidia systems under memory pressure. The problem here is likely that command submissions require a certain amount of physical memory to be available, enough to page in all buffers shared with the GPU, but the kernel driver is unable to allocate any and the application will crash, possibly without even noticing as there are no Vulkan errors in your logs. This isn't exclusive to DXVK, and often happens to me with RenderDoc whenever it runs out of system memory.

This problem gets worse if the application generates huge command buffers or uses very large amounts of shared memory inside one command buffer, but DXVK should limit data uploads to 32MB per command buffer, which really isn't all that much. The total amount of memory referenced may be significantly higher though.

I'd need an apitrace from the game to see if there's anything weird going on in DXVK, in this case it's fine to record it with DXVK as long as the game does not crash while recording it (i.e. don't use the 4GB override).

Can you also please post the full console output or anything that might show what exactly crashes?

dxvk.allowMemoryOvercommit = True

As far as I'm aware, this has no effect on Nvidia GPUs since they don't support over-allocation of device memory. It's not what we're looking for anyway since this only helps when GPUs run out of VRAM.

For the same reason, reporting a lower amount of VRAM to the application isn't going to do anything useful since you're running out of host memory, and not out of device memory. In fact, if the engine uses those limits to control texture streaming, you might even be making it worse.

Faalagorn commented 5 years ago

FYI

dxgi.maxDeviceMemory = 1024
dxgi.maxSharedMemory = 1024

doesn't actually limit VRAM usage, it just makes DXVK report that your game only has 1GB of VRAM. The game might allocate more anyway.

For the same reason, reporting a lower amount of VRAM to the application isn't going to do anything useful since you're running out of host memory, and not out of device memory. In fact, if the engine uses those limits to control texture streaming, you might even be making it worse.

Yup, I realized that too after some testing, first thinking the settings are not applied, but figured that it will use system memory after I made sure the settings are applied correctly. My friends GPU is GTX 650 Ti with 1GB GDDR5 though, so the game have to deal with only 1GB of VRAM though – thanks for replies though!

I'd need an apitrace from the game to see if there's anything weird going on in DXVK, in this case it's fine to record it with DXVK as long as the game does not crash while recording it (i.e. don't use the 4GB override).

Can you also please post the full console output or anything that might show what exactly crashes?

I will take a look at apitrace more hopefully tomorrow, as it's seems to be a little more complicated, especially since the game uses launcher, but in the meantime here are the most recent error.logs I copied, maybe they could be helpful:

  1. Unity's output_log.txt: https://gist.github.com/Faalagorn/8bdaefc638e62abf2624a48012bf4064/raw/86a1690444c9541b0e08254ceea87992680723bf/output_log.txt
  2. Wine's backtrace.txt: https://gist.github.com/Faalagorn/8bdaefc638e62abf2624a48012bf4064/raw/86a1690444c9541b0e08254ceea87992680723bf/backtrace.txt

(Also call me paranoid, but I also am a little worried about messing too much with apitrace, as the game uses a custom anti-cheat – developers said that it won't trigger bans for playing with Wine/DXVK and there's a few people including me already playing with Wine/DXVK for a while, but I'd have to make sure the anticheat is fine with apitrace for troubleshooting as well).

Thanks you anyway for the quick reply and help!

montify commented 5 years ago

I had the same issue, till i turned on "Auto Ram Cleaner" in the Game Option DXVK Version: 1.0.3 WINE Version: tkg-4.2-x86_64 GPU: gForce 970

Installed with Lutris (Runner from lutris.net) - it works fine!

Faalagorn commented 5 years ago

@montify Hm, so it does seem to be related to RAM after all I guess? Out of curiosity, how much system RAM do you have total? Is it 4GB or more?

montify commented 5 years ago

Yes, a Ram problem, but i dont know if its a dxvk or Wine issue

i have 10GB Ram

montify commented 5 years ago

So i can confirm this erorr occurs when the gpu run out of vram. i close the Webbrowser(consume ~300mb VRAM whit some tabs open) und now the problem is gone.

@doitsujin How DXVK Handle this when the gpu run out of vram?

K0bin commented 5 years ago

When you run out of VRAM DXVK allocates the resources in RAM. If Escape from Tarkov is a 32 Bit game, it may run out of address space. That's a known problem with DXVK and pretty much impossible to fix.

doitsujin commented 5 years ago

Yeah, that. With Proton and certain Wine builds (-tkg, not sure if it's in staging), you can also set WINE_LARGE_ADDRESS_AWARE=1 to double the amount of virtual memory, and DXVK 1.1 (aka latest -git) also has some code to reduce RAM usage especially in games with a large number of shaders, but 32-bit games usually don't suffer from that problem.

montify commented 5 years ago

Tarkov is a 64bit Application, so this is not the issue i guess..

I asking myself if it is a DXVK, Vulkan or Wine issue?!

I could try another "Vulkan Memory Allocator"...

Faalagorn commented 5 years ago

Yup, Escape from Tarkov it's a pure 64-bit application, including the laucher. It runs even with wine version compiled without 32-bit support (I run it like that myself) so I doubt WINE_LARGE_ADDRESS_AWARE will help, but I didn't test it since the original report, so maybe I'll see how it is with latest DXVK.

maverck commented 5 years ago

Using DXVK 1.3 produces this error on crash https://gist.github.com/maverck/4aef4dbb7b394cf3487343818eba2824

K0bin commented 5 years ago

That's the Nvidia allocation failure again. See the issue referenced above your comment for more information.

Faalagorn commented 5 years ago

I'll close this for now, observe #1100 for potential fix on the memory fix!

In the meantime, I'll try to see if the unrelated crash that's been reported in https://forum.escapefromtarkov.com/topic/1717-linux-support/ (10-ish+ pages)