MHW has performance regression

[ahjolinna]

I just tried your latest proton version (3.16-4.1-unofficial-1) with Monster Hunter: World and my performance is constantly under 30 fps, and this was even without any combat at the "lobby" area. Before I had been using the basic 3.16.6 beta version and it got in the "lobby" area ~50fps.

PS. texture loading is also little bit slower...well everything seems more sluggish...but that is maybe expected when game runs at 20fps

here is my log with VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_vktrace enabled (at the time) , if you are interested:: steam-582010.log

---

my system spec:

inxi -b
System:    Host: linux Kernel: 4.20.6-1-default x86_64 bits: 64 Desktop: KDE Plasma 5.14.5 
           Distro: openSUSE Tumbleweed 20190209 
Machine:   Type: Desktop Mobo: ASUSTeK model: Z170 PRO GAMING v: Rev X.0x serial: <root required> UEFI: American Megatrends 
           v: 3805 date: 05/16/2018 
CPU:       Quad Core: Intel Core i5-6600K type: MCP speed: 4375 MHz min/max: 800/4400 MHz 
Graphics:  Device-1: NVIDIA GM204 [GeForce GTX 970] driver: nvidia v: 410.93 
           Display: x11 server: X.Org 1.20.3 driver: nvidia resolution: 1920x1080~60Hz, 1920x1080~60Hz 
           OpenGL: renderer: GeForce GTX 970/PCIe/SSE2 v: 4.6.0 NVIDIA 410.93 
Network:   Device-1: Intel Ethernet I219-V driver: e1000e 
Drives:    Local Storage: total: 34.34 TiB used: 32.29 TiB (94.0%) 
Info:      Processes: 412 Uptime: 19:26:43  up 1 day 14:02,  4 users,  load average: 0.56, 1.16, 1.31 Memory: 15.60 GiB 
           used: 6.13 GiB (39.3%) Shell: bash inxi: 3.0.30

[kakra]

Can you reproduce any of the performance regressions with a game listed in the README so I could test and try to reproduce? MHW is not in my library...

Also, I'm seeing you're not using SCHED_ISO, neither enabled rtkit... I did not test that. But I may with one of the games listed in the README.

If possible, also try the latest NVIDIA driver (418.30) which has many Vulkan changes compared to 410.93. My version of Proton is far beyond the 3.16 base version of wine so that may make a difference.

I'm still investigating one issue that introduces micro freezes every once in a while tho I'm not sure yet where it is hiding (it wasn't there before I've lost some of my non-finalized commits to a system crash). Maybe investigating your issue could shed some light on this.

You could also try without ESYNC and see if it changes things (see user_settings.py). My build also exposes different behavior with vsync: Try turning it on or off. The vsync patch still needs some love (or may just need to be dropped if it is no longer useful after some of the latest DXVK patches).

BTW: If you're below 20 fps, it's supposed to feel sluggish.

Your log shows an exception:

136321.381:0024:002f:trace:seh:NtRaiseException code=406d1388 flags=0 addr=0x7b44e6f9 ip=7b44e6f9 tid=002f

Did anything visibly crash at some point?

[ahjolinna]

Well I can try with path of exile as it was F2P, but I don't don't have the others.. At least yet, I could buy some of them next month o ... Especially if there is nice value deal for some

About the "crash", for some reason the game won't close on proton so you have to force close it manually. From my experience it seems that it is caused by some of the graphics settings.. can't say which one yet

Btw. On that log I had vsync disabled I can check how it will run with it, also Is there any reason you think why I should disable esync.

PS. about the nvidia drivers, I use the default/official driver that openSUSE repos provides that are only LTS..so either I wait for the next LTS release (which should be soon) or I try to add the newest non-LTS driver to my repo...which would be kinda pain

[ahjolinna]

I tested MHW with esync enabled (also with vsync) ...didn't help. but after I did the "allocstall" thing that seems to have made it run smoother(less drops?)....even if the fps are still under 30.

Btw. The FPS is now usually at 25-28 but there's one stress area that I tested which dipped to below 25fps with the basic proton but with your version I got the same 25-28fps

~~here is the latest MHW log, without esync (but with vsync and allocstall): steam-582010.log and with esync: steam-582010.log~~

I really need too sleep.. more often and maybe reboot my PC also at times..... apparently MHW runs now as before, I think it was after the "allocstall" thing and a reboot :D -_- esync didn't have any extra effect at least, but enabling v-sync seems to had some positive impact....well minor but still. here is the log where is test the v-sync in one good stress area steam-582010.log

I also tested PoE, and it runs well and apparently out of the box...which is great, only really minor stuttering and fps drops, this is with all settings turned to max. There is the log for it: steam-238960.log

it seems at least PoE is still the same after my reboot, but here is still my latest latest log after the reboot: steam-238960.log

There is FF XIV (mmorpg) that I will test next asp when their servers are out from maintenance..as it also uses DX11. FFXIV also has a free trial version up to level 35, and apparently uses the same exact launcher also so it's perfect for free testing. (there are at least some game file conf's that needs to be changed before playing, like it has been said here)

PS. I'm currently working on adding the latest (beta-LTS) nvidia driver to my repo when I get it to work I will tell the results

[ahjolinna]

btw. I wanted to mention the reason why I'm testing your proton version is mainly to check if the this issue https://github.com/doitsujin/dxvk/issues/816 for MHW was still relevant and if there was any performance improvements

[ahjolinna]

I tested FFXIV, and I get 50-60fps with all settings maxed ....there is some performance improvements if you are interested here is a "small" log (70mb) : steam-39210.log

[kakra]

Many GPU hangs with XID messages in dmesg can be fixed by updating the NVIDIA driver to fully support the used Vulkan extensions properly. GPU hangs usually resolve after 20-60s when the graphics system resets itself but you should probably still SSH into the system and kill the responsible process. Your freezes may well result from such problems.

I'm gonna look at your logs later, I managed to kill my system again by overloading it while unattended. I probably need to restore from backup. There's some nasty bug in the kernel when handling memory-pressure situations, probably in bcache (bcache lost a series of transactions mid-air and only noticed after reboot)... :-(

So, can we recap?

• FFXIV: Not a performance regression after all, performance even improved?
• PoE: Mostly unchanged?
• MHW: Still clueless?
• Does the freeze on exit still occur? (it looks similar to what can be seen in Skyrim, it's related to xaudio somehow)
• Is NVAPI still a problem with FFXIV and DXVK? Because I'm planning on putting it back in with wine-4.2...

[ahjolinna]

well I would maybe SSH my PC if I had an extra PC/laptop, but I just borrowed mine ultrabook to my mother ....maybe I could use my good old trusty Nokia N900 :D (I have it somewhere).

oh and maybe stop torturing your PC that much, or have if you really need to...maybe have two distros/systems: one for bleeding edge and torturing, the other for more stable and backup usage (I know it's boring but they are good control units)

about the recap

• YES
• YES
• kinda, (like I said performance was now in "normal" level after "allocstall" thing and a reboot + vsync)
• the freeze issue is still "don't know yet", as that problem is so bloody random, it has nothing to with exiting the game, as it does exit just fine if you logout (saves) first then exit ...so I think that problem has something to do with autosave.
• what NVAPI problem? (if you mean that you need to do game config workaround for it work*)

_*I'm talking about these two files FFXIVBoot.cfg and FFXIV.cfg

[ahjolinna]

well for now I haven't got any of the random "freeze" issues yet (if any idea what would trigger it, do tell)....and even existing the game seems to work ...at least if you have saved just before you exited...have to double check if this is the case, anyway here is the latest log of MHW: steam-582010.log

btw. I would hope (main)distros will make more needed adjustment or at least a easy for enable those setting for mainstream users... if some of your tweaks, patches, hacks & conf. gets to the upstream releases...at least that "allocstall" thing.

I would love to have something like the ck kernel on openSUSE, but that is such a pain to maintain by yourself....yes I could take the basic openSUSE kernel spec/conf and then modify it...question is how do I need to modify...if not much, then it would be easier to maintain

[ahjolinna]

apparently valve just released proton 3.16-7, are you going to release a new version based on that? ...not sure if its using wine4.x as it doesn't say changelogs...so maybe not, I wonder what is taking their time (if not)

[kakra]

The Proton version always identifies the Wine version it is made from. Proton 3.16 will always be Wine 3.16. My version rebases it to a newer Wine version, adding selected staging patches and adding some of my own patches (including subprojects like DXVK). But thanks for the notification, I'll look at the new patches in the new patchlevel version of Proton and cherry-pick them into my branch.

@ahjolinna About torturing my PC: I'm actually having a second minimal system on a flash drive as a rescue system to boot from, including restore/repair tools as needed. A full system backup runs daily using borg backup to store a backup to a NAS. I loose at most one day of work, I prevent that by pushing my git repositories to the same NAS after I made some bigger changes. So I did loose maybe one or two easy commits which I can recreate easily. But the restore takes at least 24 hours, and since I moved the borg repository to NFS, it seems to run a restore at only half the speed - so I expect at least 48 hours. My backup uses a retention policy to keep up to 5 years of backups, currently around 100 full system images stored deduplicated (with a ratio of around 98:1 currently). So even if I delete something, I can currently go back over 2 years to restore from.

Last thing I did was updating the system with emerge (this included packages like OpenOffice and WebKit which require a huge amount of RAM to build), I started a clean wine rebuild of my system at the same time and a kernel rebuild, just to leave it running over night. Next day, the system ran just fine except the KDE session seems to have restarted. Looking at dmesg, I found that systemd-journal exited and restarted, this is almost always a sign of memory pressure. There were no traces in dmesg, so I just rebooted - bang. The backup also didn't start that night which set me back another day.

The first backup after restore will take something between 15-20 hours (successive runs only take 0.5-1 hour) during which I can do only moderate work with the system. Another 20 hours or so are needed for deduplication to catch-up again (I do not bother much to not store duplicate files in my system, I let bees to the work for me, that's mostly files I sync between different machines with syncthing). So all of my files are stored at least twice with important ones on different machines. I think this is a backup strategy you could agree with. ;-)

[ahjolinna]

well at least you have a proper setup it seems...with some bad luck apparently "c'est la vie" anyways, I was wondering have you thought about using OBS to build your proton build? you can make your own setup/server or just use openSUSE one. Its really handy pkgs build tool/service that works for "everyone"

PS. what do you think about this project, are there anything you could add to yours https://github.com/popsUlfr/Proton also do u think this could be somehow integrated? https://github.com/simons-public/protonfixes

[kakra]

@ahjolinna Yeah, I think there's a bug hiding somewhere in bcache when running under memory pressure. The problem is much more easy to hit when you enable zswap because there's more memory allocations on the hot path then. And I've seen similar reports with similar corruption lately by looking at the btrfs list. After my system is back, I'm going to report this.

Regarding OBS, I'll look into it. But I thought it is only about distribution packaging? This is really not the way how this is distributed. PS: I'm working a universal Proton launcher which enables you to configure various Proton versions and layering of Proton binary patches per game. It would be interesting to see if OBS could be of any benefit here. It would also enable you to use protonfixes (or a very similar approach).

I'm already watching the popsUlfr repository, but currently it provides no benefit to my repo or my system. But it is already part of the previously mentioned launcher as an example for adding more Proton versions.

[kakra]

This shouldn't have been closed... Yeah, fun again with foreign issue numbers... \o/

[ahjolinna]

I'm not sure If I should make yet another separate bug report, but... I just tested Path of Exile with your latest version (3.16-4.2-unofficial-1) with the latest nvidia driver (418.43) ...the performance/fps went now down to 40fps at times...and even started to freeze for few seconds, anyways here is the log steam-238960.log

[ahjolinna]

and with MHW, it froze and crashed....I think stream crashed first*...anyway...here is the log: steam-582010.log

I'm not 100% sure which one crashed first but I got this error from dmesg ;: [ 1870.001155] steam[8979]: segfault at 5 ip 00000000f029d991 sp 00000000ffc28470 error 4 in libdbus-1.so.3.5.8[f0289000+47000]

[kakra]

I did only a quick test and PoE played fine. You shouldn't use 418.43 but 418.31.03 which has more vulkan extensions. Could you try that driver?

But OTOH, my setup is currently a little limited, only one 4k monitor, no dual monitor setup. So I don't currently compare performance. I was running PoE on the 4k monitor previously and found no difference. All other games are running stretched to 4k, and that looses up to 30% of FPS in some games.

Segfaults in Steam will often crash a game. But it's probably not a crash in the game itself. Looking at your logs this is what happened: Pipe to Steam API was closed, and then the game crashed because it doesn't handle that situation. It seems unrelated.

PoE may have to rebuild the pipeline cache. Play for at least 20 minutes by walking through different maps. It should settle down after a while. Then repeat the same map twice. Do you still see performance problems that last time?

[ahjolinna]

for now I will test only using the latest LTS driver as I like to use/test stable stuff, I think I may do an arch installation where I do all the more "bleeding edge" testing: anyways, PoE seems to work now normally on the 2nd try ....still the stupid stutters but same as before..here is the latest log (if you want): steam-238960.log and MWH seems to work again with the 2nd try...don't know why I had this issue the first try...oh well, here is the latest log steam-582010.log

[kakra]

When you experience slow performance, could you compare the output of nvidia-smi after quitting the game versus when performance was normal (i.e. after a fresh reboot)? I've seen performance problems gone after a fresh reboot. It was almost always related to different processes eating huge amounts of GPU memory. Performance will drop again when you fill GPU memory (e.g. by spawning a lot of Google Chrome tabs). The underlying problem is bad memory management in the NVIDIA driver: Currently, it cannot migrate cold memory allocations from device memory to system memory. Maybe that's your issue, too? A work-around that might work is lowering the texture resolution in the game (and going down just one step, i.e. from ultra to high, usually cuts memory usage in half while having almost no effect on rendering quality).

If device memory is exhausted, DXVK may allocate texture memory from system memory only, the GPU then starts rendering textures from system memory: It's a lot slower (usually by factor 2 to 3), and it may even become unstable because device-local-only allocations may fail (often resulting in a freeze or crash of the game engine).

BTW: I downloaded - but did not test yet - the demo of FFXIV. Usually, I don't like JRPGs that much - let's see. ;-) Valve's upstream bug report mentioned NVAPI-related performance problems (which I mentioned above). Will take some days... Stay tuned.

[ahjolinna]

yes this was after fresh reboot, and I will check nvidia-smi next time...also I wasn't yet that "worried" about the issue I just wanted to report it just in case as long as was still awake and I had the logs saved.

about FFXIV its nice that it has a free trial until lvl 35 so it's good for testing and as it's really popular mmorpg with like ~14M players (subscribed ones) so it's a good thing to have it working well on linux/proton...

also, MHW is one great game that sold really well (Capcom said that it sold like 5M alone for PS4 in japan/asian only) and it hasn't any microtransaction or loobox BS, but still has frequent free DLC/content updates ..MHW is also my GOTY (2018) for PC (I have also it for my PS4pro), I personally can't wait for the iceborne expansion..here is also my fav review

next game I'm exited for is Square Enix: Left Alive (I bought the mech edition) for PS4 and PC, coming march 5th....I did order the PS4 version as I don't like to take change that it wouldn't work on proton... (as I only use Linux) and if it does work I will buy it later when on steam when it's really cheap :D ...if interested here is a trailer

anyway enough with the gaming promo talk :P I'm guessing you are familiar with DXVK-native project?

(o^^)o PS: yay, DXVK v1.0 has arrived (fixes crashes issue on MHW with nvidia 418.43 driver...apparently) and faudio re-implementation for wine has landed ..it's nice to see some progress...even if its slow at times o(^^o)

[ahjolinna]

@kakra : just small update inquiry, how is your system doing and this project? also valve just released proton 3.16-8 with DXVK 1.0 + other minor fixes (maybe look at them)... for some reason Valve hasn't yet transitioned to Wine 4.x but hopefully we'll see them re-base soon.

[kakra]

Please only carefully compare this wine version to Proton (which consists of wine, lsteamclient, and more). DXVK is almost always the latest master version, so with the next version I will be ahead of 1.0 compared to Proton. The wine part of Proton only has some minor backports which are already in my wine version because I rebased to a newer wine. There may be one commit missing compared to Proton, I'll look into it with current release.

Actually, looking at my rebase, there's not many commits left from Proton upstream, mostly full-screen hacks and Steam API hacks. The majority of other commits has been upstreamed to official wine. So the vast amount of commits in my branch comes from staging-patches or custom patches I did. I think a 4.x rebase of Proton officially shouldn't be too far away.

But you already figured out that some games do not work with my branch which work with Proton-official. Maybe upstream wine introduced some regressions (or I did my rebases wrong). So this may be why official 4.x rebase is still blocked.

My system itself is running fine so far (I managed to rescue it and all devices, including the backup, it's a physical problem not a computer problem) but I needed to move it to a different location temporarily (including some limitations like single-monitor only and low internet bandwidth) which doesn't allow me to invest as much spare time currently. This situation will probably last 3-6 months from now. I'll keep the branch updated while arranging with the situation better (or changing it completely) but I'm going to lag behind. I'm currently preparing the 4.3 update but FAudio changed its packaging so I need to adjust my build system. Stay tuned, I'm planning an update for this weekend.

[ahjolinna]

yes I do understand that is a mostly different from proton, but why I mentioned the latest update was just because there was some extra fixes and to inform about them. Also it's good to know that you use more bleeding edge version of dxvk...maybe someday you could have both stable and "git/dev" version? (when your personal life is getting better)

about wine/proton, its nice to see that many fixes/"hacks" have been upstreamed, I do hope wine-staging would become way smaller than ~800 patches. it's also good news that proton & dxvk have started to become quite good & more matured projects and with wine now "expanding" thanks to this it will allow more interesting experiments for projects like this.

single monitor + shit internet (not less than 100M I hope)...omg, ouch, well "c'est la vie" I guess...good luck with your endeavours.

[kakra]

@ahjolinna Well, I had 1G/s before, now it's more like 50M. That's quite a throttle. BTW: I'm currently uploading the 4.3 rebase...

[ahjolinna]

okay not as bad, I just myself upgraded to 1G/s (upload and download, unlimited usage for 60€/month)...

Anyway I'm will test your latest version, I also just bought some more games to test "the humble neptunia bundle" (just endend) for ~14€ and Sleeping Dogs: Definitive Edition for ~5€...and apparently some are in the bronze/broken

so I was thinking, if MHW works now just fine now we/I could close this bug report. And I will start to make a new reports of other games (seperate?) if needed and if there is some new problems with MHW, I will open another one.

[ahjolinna]

okay MHW seems to run little bit smoother than before, but it's sad that old the freezing issue still exist (nobody hasn't figuredout where the problem lies)...and it lags the whole system up

at least doesn't use all GPU memory

gpustat -P --show-cmd
linux                Sun Mar 10 07:48:15 2019  418.43
[0] GeForce GTX 970  | 64'C,   0 %,   70 / 201 W |  3096 /  4034 MB | Z:\home\ahjolinna\.local\share\Steam\steamapps\common\Monster Hunter World\MonsterHunterWorld.exe(2377M) X(346M) krunner(16M) kwin_x11(80M) plasmashell(141M) steam(33M) steamwebhelper(3M) DiscordCanary --type=gpu-process --enable-features=SharedArrayBuffer --no-sandbox --gpu-preferences=KAAAAAAAAACAAABAAQAAAAAAAAAAAGAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=14519051481850857879(40M) Z:\home\ahjolinna\.local\share\Steam\steamapps\common\Monster Hunter World\MonsterHunterWorld.exe(2377M) ksysguard(3M)

and cpu usage never goes over 50%

well here is the log: steam-582010.log

I also have nvidia-bug-report.log two versions, pre and post reboot

older/pre-reboot log & post reboot log (+mhw testing)

it's up to you if we should make another bug report about this and close this

[kakra]

What's the freezing issue? Maybe it makes sense to create one issue per problem, then list the affected games there. That way, when a game has been identified to match one exact issue, the game-specific issue could be closed. It could also make sense to document work-arounds and findings in a wiki instead.

[ahjolinna]

its the old random issue (doitsujin/dxvk/issues/816)

[kakra]

Interestingly, in my single-monitor setup (which means the GPU driver can use flip instead of blit) I'm running constantly below 30 fps in Shadow of the Tomb Raider. Something seems to be different in my multi-monitor setup (which I cannot use currently), and although flip should be faster than blit, it's still below 30 fps. I think this is related to vsync somehow which behaves differently in a multi-monitor setup.

[ahjolinna]

is that issue with DX11 or DX12 as it has both

[ahjolinna]

small update, as nvidia just released their new vulkan driver (418.49.04), I tried MHW with it and using your latest build. now it runs way smoother even with almost all settings at max, I don't get below 30fps anymore (usually it went down to 20fps).

and for now it hasn't frozen yet....it almost did at the beginning of testing, but just for few seconds, anyway if you are interested here is the latest log: steam-582010.log

PS. there is still the "force close" issue, don't know why it won't close down normally on proton

[ahjolinna]

okay still freezes with the latest 418.49.04 "vulkan driver", here is the log if interested: steam-582010.log PS. I dont think there is any need to reopen this bug report as there is no regression anymore, if you want I can make another new one about the freezing and closing issue (which affect proton)

[ahjolinna]

I have had now few good issue free play times, even closing worked now (no need to force exit) ..but not according to the log? : steam-582010.log

PS. valve just release proton v4.2 (maybe check what their changes):

well MHW did freeze quite fast with that v4.2: steam-582010.log

apparently proton has 214 patch on top of wine 4.2