Proton Experimental: Multiple games experience input lag / input event loss

[kakra]

Using latest Proton (probably started with experimental-wine-5.13-20210120 or experimental-wine-5.13-20210115), multiple games see input event loss or extreme lag.

Usually, single button presses on the gamepad aren't recognized, pressing it a second or third time and it works.

Sometimes, button presses are detected but the release isn't detected.

Axes sometimes do not update and just hang in some position until the situation eventually recovers after some time.

Button presses sometimes lag behind (usually just a short period, sometimes seconds). Axes also lag behind, often releasing an axis lags behind several seconds (see above).

In some games, this can also be observed for keyboard events. Mouse movement seems not to be affected but mouse buttons are. I usually do not play with mouse so I cannot really confirm this.

Games tested and affected:

• Borderlands 3
• Detroit: Become Human (this is very extreme)
• Control (by Remedy)

The problem is probably depending on CPU usage of the game, titles putting more pressure on the CPU experience the problem more intense. From a programmer perspective, I'd say that the scheduler is starving the input event threads in wine. This MAY be a result from using the MuQSS scheduler in my kernel but even if it is, this is a problem that should be fixed in wine as scheduler behavior tends to change over time or due to pressure put on the system.

[rbernon]

Thanks for the report, are you using rtkit-daemon or configured ulimit so that the new priority mechanism work? Could you try with that disabled, by uninstalling rtkit-daemon and have 0 RLIMIT_NICE?

Same question with fsync / futex2, it would be nice if we can narrow this down a bit more.

[rbernon]

I'm guessing this could be related to https://github.com/ValveSoftware/wine/commit/287781d90fff4b22dbed805b2a879ffa5ccbeafa#commitcomment-45900927, or some similar issue with driver processes that don't have any specific priority set and could so lag behind the game threads.

[kakra]

Yes, I'm actually using rtkit-daemon and configured ulimit for -15. I used similar patches in my custom-baked Proton back in version 3.x and 4.x but didn't experience such behavior there. Actually, I had one patch that increased priority of the input-related threads/drivers - not sure how wine (or Proton for that matter) changed since then. I'm also using a fsync / futex2 patched kernel (including MuQSS) but since multiple games show problems starting with futex2, I disabled it globally by a session environment variable.

[kakra]

BTW: I recently added a patchset for bcache which removes a system_wq workqueue congestion from the kernel, and it looks like this lag is no longer an issue in Wine/Proton. But it shows there's still a potential to get this which may somehow be related to the kernel event handler (system_wq): https://lore.kernel.org/linux-bcache/CAC2ZOYtrrOXD35ZWLer8R8n1dcyqGi_dipHp+y1JNt_eaue_9A@mail.gmail.com/T/#t

This kernel patch series is proposed for inclusion in 5.12 with backports back to 5.4+. I'm currently applying that on top of my 5.10 kernel.

[kakra]

These are my input driver patches I added to Proton:

• https://github.com/kakra/wine-proton/commit/6e2f524ac5f72f7eec416faf89af8ecc4982aa94
• https://github.com/kakra/wine-proton/commit/68c0a024c8ce47b440b817125b59127365e31010

I added these because I was seeing slow hotplug input device detection in demanding games, and missed input events (back when I played Tomb Raider, the trigger button often didn't engage the bow). It did improve things but didn't fix it. In the aftermath, that may have also been an instance of the system_wq congestion which has been introduced into bcache in 2014.

[CSahajdacny]

It will be useful to have a dedicated Proton Experimental thread for to report regressions.

[kakra]

@rbernon

are you using rtkit-daemon or configured ulimit so that the new priority mechanism work? Could you try with that disabled

I just found that I actually didn't enable that service for some unknown reason, so it has been inactive anyways. I'm going to observe the behavior for some time with it enabled.

Ignore that, it auto-spawns on dbus calls. Usually after system boot it had been running because some process triggered it - seems to happen no longer and thus I assumed it wasn't working at all. But looking back at older logs, it spawns when I play Proton games.

[kakra]

@rbernon Do you know if it would be enough to make all winedevice.exe threads highest priority for an artificial test? Actually, I did that (alt+tab out of the game, run schedutil -I -n -19 $(pgrep -w winedevice), cross-checked in htop that priority was changed successfully). But it didn't help the issue.

I'm not sure why that happens: Sometimes it just works, sometime the problem persists until I eventually reboot the system. While the game is running (and using CPU), I ran jstest and evtest to validate that the kernel sees input instantly: So lag doesn't come from the kernel.

I also disabled vsync and triple buffering to verify that I'm not just seeing delayed screen content.

The issue is currently different to why I initially reported it. I think my initial issue is actually some Bluetooth problem with my Elite 2 controller, I don't see that (delays of multiple seconds, input event loss) happen with my Xbox Series X controller. But there's still a constant lag of what feels like 200ms across all games I'm testing. In a fast paced shooter this feels like running around while drunk - somewhat funny but not very useful without actually being drunk. ;-)

Enabling futex2 only seems to make that worse (but the whole implementation seems to not work correctly for me, I get audio-dropouts and inconsistent frame rates when using futex2, so that's probably a different issue).

Mouse and keyboard input experiences the same lag, so it's not specific to the driver.

[kakra]

It looks like vkd3d games don't experience this issue...

[rbernon]

Mouse and keyboard input is usually received by the application process itself, in the thread(s) that call Peek/GetMessage, and then there's a roundtrip to wineserver. The winedevice processes are only receiving gamepad input so far.

One exception for mouse input though, if the game uses rawinput, then some of the input is received in explorer.exe, which pushes it to wineserver, which in turn dispatches to the right application threads. In this case it will also depend on the application thread (possibly dinput thread if the game uses dinput) calling Peek/GetMessage.

Among these I only see explorer.exe, wineserver and the dinput thread for which increasing priority could make a difference. I don't think we should change the priority of the application threads themselves, but if they have a low priority and for some reason don't call Peek/GetMessage often enough, then it could be an issue.

[rbernon]

Ah so I didn't re-read the initial message and I see now this was about gamepad input. So in this case the flow is indeed coming from winedevice.exe, which replies to (wine) ioctls, or pushes input messages to wineserver. In all cases there's wineserver involved in the middle, and ioctls / device read (used by xinput I think) is currently the worst as it involves multiple roundtrips from the application to wineserver and from winedevice.exe to wineserver.

[kakra]

Is this true for SDL patched Proton? As far as I could track it, winebus.sys directly interfaces with SDL, but I'm not sure how that integrates with xinput. Maybe I have to get back into some Proton development and testing patches... From your description, whatever winebus.sys does, lives in a winedevice.exe thread - and passing information between the game and winebus.sys has to go through wineserver? This sounds somewhat crazy to do. I can just guess this is about abstraction across different platforms but Proton is quite specifically for Windows games on Linux. Couldn't we just call from xinput right into the Linux APIs?

[rbernon]

From your description, whatever winebus.sys does, lives in a winedevice.exe thread - and passing information between the game and winebus.sys has to go through wineserver?

Yes, that's correct.

Couldn't we just call from xinput right into the Linux APIs?

It possibly could for simple input but I think it would fail to work correctly in some cases, or be much more difficult to implement, for all the Windows internal interactions to work properly and consistently across processes.

It's clearly not ideal, and especially with the current overhead of things like reading HID reports from device files, but there are ways to make it better, it only needs time and effort to implement the required mechanisms.

[kakra]

I wonder if things could be improved if wineserver wouldn't be strictly single threaded: It could implement different threads to serialize independent subsystems individually instead of putting all and everything on the same queue. But that's probably not easy to implement either, maybe even impossible.

[Laxystem]

Can confirm for Slime Rancher 2, input lag of whole minutes.

Doesn't happen on Proton GE 8-32, tho. Does happen, but a lot less.

[codymikol]

Happening for me now on a bunch of proton games, the longer I play the worse it gets. I was playing subnautica and it got to the point where input delays were around 30 seconds.

Edit: Looks like https://github.com/ValveSoftware/Proton/issues/5294#issuecomment-1236124959 is relevant, going to see if this fixes my issue. I think my FPS is just dropping over time for some reason and then the problem in this thread becomes relevant.

[kakra]

Is this on wayland? If it is, it is a different issue from my initial post.

[codymikol]

It was actually my aio cooler breaking lol