Closed Logical-sh closed 3 years ago
The xpadneo rumble code should not be much different from what the wired driver uses. But there are a few differences:
In wireless mode, we are limited to updating rumble at a rate of 100 Hz maximum, any faster updates will crash the controller. But this timing resolution should be fast enough to not notice any difference.
What's the wired driver you're using? (it won't be xpadneo) Depending on the driver, it may not use ff-memless force feedback driver. ff-memless is a force feedback driver which emulates force feedback commands through rumble commands, Its internal update frequency is 50 Hz. This may leave an impression of slightly longer rumble than you're used to. It's still near the frame render times of games so it usually shouldn't make any difference.
Since you're on kernel 5.12, try enabling ERTM. The current git master should have a detection for kernel 5.12 to not disable ERTM but there may be some left over file in the module options from previous installs. ERTM has an impact on how stable sending rumble commands is. I've yet to test if we can remove the 100 Hz rumble throttle with ERTM enabled.
Please look at dmesg output to check whether the game triggers rumble throttling when it starts rumbling. I'm not sure why this is needed at all because ff-memless is supposed to send rumble commands in intervals of at least 20ms but if the driver is asked to send at intervals shorter than 10ms, we will throttle rumble. The message is only logged once per module load to not spam dmesg. So you may want to start with the module freshly reloaded.
Also, do you see any lag with input events while you're gaming? The firmware has some problems with Bluetooth interference, especially while BT inquiries are running. If you're using KDE, or have kdeconnect installed, try removing kdeconnect temporarily as a test.
I'm currently evaluating if we should remove ff-memless support from xpadneo as no games seems to use special rumble effects like sawtooth, sine, or square waves - this would give games more direct control over rumble with less lagging behind (the lag depends on your kernel HZ setting, a 1000 Hz kernel could lag 1-2 ms behind, a 500 Hz kernel could lag 2-4 ms behind, it's not much but if you use a 100 Hz kernel which you probably don't, that would be 10-20 ms).
Do you know of other more modern games I could test with, and which show the same behavior for you?
What do you mean by "drawn out"? Many drivers program the wrong rumble strength scale, they actually allow rumble strength from 0-255 while the controller only supports 0-100 (this is stated in both MS docs and the HID descriptor, sending values above 100 will just rumble at 100% strength, which may result in a "drawn out" feeling because xpadneo does it correctly and other drivers rumble "too strong").
I am using xpad I think when wired (I don't have any special driver installed, but I think steam may use custom handling, not sure?)
I looked at xpad, it uses the same ff-memless registration for rumble as xpadneo does. But it uses a wrong scale for rumble: it scales Linux FF values 0..65535 to rumble values 0..255 while the gamepad only supports 0..100. Thus, you'll experience the same lag and effect combination than xpadneo but 2.5x stronger with a peak for all values above 100. As a result, effects with xpadneo are more soft and subtle while xpad has quite strong and hard effects. Additionally, you may become confused because xpadneo has some internal magic to make the triggers rumble, too: It uses the main motors magnitude to compute a rumble strength from it and how far the trigger was pressed. Linux rumble protocol has no way to handle more than two rumble motors, so I became creative.
Regarding Steam, that's a different story: Steam has it's own mapping feature which integrates with SDL2 (and may also impact rumble but I'm not sure). SDL2 may use its own rumble driver which would completely bypass ff-memless. I currently do not know if we could easily prevent SDL2 from running its own rumble driver for USB-connected gamepads. You could probably try revoking access bits from the device node file that xpad creates - but I don't know what that device node is called because I used xpad only some years ago.
Additionally, you could try disabling Steam input for your controller to cross-check. It won't have any impact on the SDL2 rumble driver, tho: That would still be used.
I did enable ERTM! The stability and reliability of the connection is greatly improved, this controller was unusable before!
Thanks for confirming.
It is triggering rumble throttling, but I am not sire that is the actual whole issue here, I changed the delay to 1ms instead of 10, and the controller performed fine, but still seemed to have this issue.
So you changed the rumble throttle to 1ms and didn't see any issues with ERTM enabled? That would be nice. But I'm not sure if we can yet confirm this. For one, you should be able to make rumble crash the controller when lowering that to 1ms, and xow (which uses the dongle that comes with the controller) shows the same 10ms problem although it doesn't use Bluetooth - that would mean ERTM should not have an effect on this.
I don't seem to have lag issues, at least not that my currently limited testing has noticed, this rumble issue is hard to play with, I will explain better below.
Yeah, it's always better to experience that yourself. Let's see if we can find a game which I could test with.
I have noticed it in Assasin's Creed 2 and The Surge 2 (A dark souls like scifi game)
I don't own either of these games. I own AC:{Black Flag,Unity,Origins,Odyssey}
and Far Cry {3,4}
.
I think I figured out a better way to describe this. Its almost like all the rumble effects get queued up to play one after another instead of new input overwriting the current effect.
Well, that could be explained when ff-memless is used because it is able to stack and combine effects instead of overwriting whatever effect was played before. But when SDL2 uses its own rumble driver for your gamepad, it would bypass ff-memless because it won't program effects through the driver but directly through the Linux device node. Revoking access from such a device node could force SDL2 to use the event driver. I'm not sure if this will work for xpad, tho, as that is not a HID driver. You could run evtest
to see if the xpad driver generates input events through the input layer.
With ff-memless, effects will play with a loop count and rumble period parameter, successive effects will we stacked on top of already playing effects until their effect duration times out individually. Sending a null effect to stop rumble would do nothing because a null effect would stack onto playing effects. This will be different without ff-memless: Sending a null effect would replace any current effect, effectively stopping it. This could explain what you experience.
(Not 100% sure this explanation is completely accurate, but is close enough to explain the issue I think.)
I think I get what you mean.
When playing the surge 2, it rumbles when you block, or hit an enemy. When wired, if you hit the enemy 10 times in rapid succession, it will stop rumbling right after the 10th hit. So if for example it would rumble for 1 second on a hit, 1 second after the last hit all rumble stops.
A duration of 1s should be just fine, no matter if ff-memless is used or isn't. This somehow doesn't support my theory: Effects would be spaced out enough to not overlap.
When wireless its like it plays the 10 seconds of rumbles even if all ten hits happened in less than a second.
This should not happen: Effects can play for well below 100ms for me in games. Also, effects are not queued, they are played promptly and may be stacked ontop of each other (by adding the rumble strengths of parallel effects). So I think you're actually seeing one of the Bluetooth bugs of the controller where it processes commands too slowly due to Bluetooth issues. Upgrading the controller firmware may solve the issue, using a different Bluetooth dongle may also work around it.
The result is that if I finish a rapid fight with an enemy, the controller continues to rumble as if I was still fighting for many seconds after the fight has finished and I have entered menus or something.
This more sounds like an issue I had when rumble throttling wasn't implemented: The gamepad would sometimes continue to rumble for a few seconds with a static non-changing effect before finally stopping. Instead of reducing rumble throttle timeout, could you try increasing it to 20ms or 50ms? Don't forget to reload the module after recompiling it.
As mentioned above, I tried reducing the delay from 10 ms to 1 in the header file, and that removed the throttling but this issue still persisted.
The issue may actually not be too long throttling but too short throttling which ends up with the firmware missing some rumble events or partially crashing. increasing the timeout may prevent that (OTOH, effect updates will lag behind longer but using 20ms or 50ms should still be short enough to do your 1 hit/second test). If you can confirm this, your kernel timing may be somewhat off or strange. There are bugs in Proton currently which affects timing for certain processors when adaptive CPU frequencies are used: Setting the CPU to performance mode may work around that. Also, with timing off a little bit and using a high-HZ kernel (1000 HZ), we may actually run below the 10ms throttle sometimes. Setting it to 15ms or 20ms may work around that.
My way of testing the change was to uninstall.sh, rmmod, disconnect, install.sh, reconnect. what I meant by around the issue was that I no longer got a log line that it was throttling anymore. I can record audio or something of the difference with the controller held to the table if that might be clearer than my explanation? I wish there was an easy way to record just the rumble effects sent to the controller! But I think you may be right about the steam thing, I can see if I can get it to run wired without using steam input libs, then that would point to a difference in the way the ff-memless handles rumble compared to the way steam/windows handles it if I am understanding that correctly?
As far as the games you have, AC Black flag would be the closest in engine to AC 2 I believe (thinking age and dev teams) so if you wanted to see if you could find a difference, that's what I might try. If you didn't mind sharing your steam username I could compare our libraries and see if I could find some games we both have as well, if it needs to go that route, but it is sounding like this might be a ff-memless handling difference as opposed to a xpadneo issue...
I don't think the 1ms caused issues with the controller firmware (input was fine during the odd rumbles before ERTM it would just disconnect within minutes if I even got it working.)
I will start by seeing if I can replicate the issue wired when steam input is disabled.
OK so observations, I found a move that reliably triggers this oddity, A basic charged attack. -Steam input Forced Off (xpad confimed by rmmod disconnecting controller.): -- Short vibration on charge, longer vibration on release. Rather muted -Steam Input Forced On: -- Exactly the same. -xpadneo -- 5ish lighter pulses on charge, 6+ heavier pulses on release.
I don't think Steam input changes the rumble behavior on its own (except that you may be able to adjust the strength) - SDL2 may do that. SDL2 in raw device mode will replace the rumble driver, it should support both modes for xpad, it only supports evdev mode for xpadneo (we currently disable raw mode due to wrong button mappings in that mode). Switching Steam input won't change if ff-memless is used or isn't - it's only that SDL2 in raw mode cannot use ff-memless, only evdev mode does if the driver supports it (both xpad and xpadneo use ff-memless).
Steam: https://steamcommunity.com/id/hurikhan77/games/?tab=all
But I'm currently most convinced that this is neither an issue with ff-memless nor an issue with xpadneo: If ff-memless would be the problem, you would have seen the issue before when Steam runtime still shipped with an older SDL2 version (pre 2.12), and xpadneo does nothing different and xpad in this regard. I think the rumble programming over Bluetooth is broken. There's a test app in misc/examples/c_hidraw
which allows programming bypassing the evdev driver (which sits above xpad and xpadneo thus also bypassing ff-memless), it actually can program the rumble like SDL2 can in raw mode. But you probably won't be able to trigger the bug there because you cannot create events fast enough as the game does. But you can try by programming an effect and then holding enter to send the command over and over again.
I added event logging to the module in xpadneo_ff_play
pr_info("xpadneo - debug - Effect: %d - Dir: %u - Replay: [d:%u l:%u] - Rumble: [w:%u s:%u]", effect->id,
effect->direction,
effect->replay.delay,
effect->replay.length,
effect->u.rumble.weak_magnitude, effect->u.rumble.strong_magnitude);
Results in a log like this
[195650.783984] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.784663] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.800279] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195650.800286] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.804662] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.804667] xpadneo 0005:045E:0B13.007E: throttling rumble reprogramming
[195650.816992] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195650.816997] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.821335] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.833550] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195650.833557] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.837993] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.850311] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195650.850318] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.854661] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:32512 s:0]
[195650.867007] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.368460] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.374660] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.385132] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.385138] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.387998] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.401777] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.401783] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.404663] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.418400] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.418407] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.421331] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.435242] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.435248] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.437997] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.451975] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.451983] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.454664] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.468416] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.468423] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.471330] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.485170] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.485176] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.487997] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.501711] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.501717] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.504664] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.518238] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.518244] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.521330] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.534977] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.534982] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.537997] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.551705] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.551712] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.554664] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.568251] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:0 s:0]
[195652.568257] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
[195652.571330] xpadneo - debug - Effect: 0 - Dir: 0 - Replay: [d:0 l:0] - Rumble: [w:44544 s:26112]
It looks like the rumble is being turned off and back on again several time in the span of less than a second... I am very new to device development. haha. I can take a look at that example and see if I can figure something out. I didn't realize wine used SDL for input, Would enabling raw mode for SDL be a useful test? I can trigger the effect pretty easily even if SDL mappings are not right I think.
You'd need to revert this change to enable hidraw support for SDL2: https://github.com/atar-axis/xpadneo/commit/f7e1e8a9ac7fddb1221df6b46a5a774dba94b95d
I didn't realize wine used SDL for input, Would enabling raw mode for SDL be a useful test?
Wine doesn't use SDL2 but Proton-patched wine does for supporting controllers directly or through Steam input properly (it has a special code path for identifying and handling controllers passed through Steam input), actually that's part of Valve's effort to emulate all controllers as xinput controllers in Proton, so games would see a Xbox gamepad no matter what it is. Enabling raw mode so SDL can use it would provide a testing pattern that bypasses ff-memless. You could try if you still see the same behavior then. You can enable Proton logging to confirm this (I've put links below which give you some insights on how to properly log it).
Excessive logging to dmesg can slow down the system considerably... That may change the bug behavior. also you may rather log xpadneo_ff_worker as that is actually sending commands to the controller. xpadneo_ff_play is just ff-memless calling into us. I just realize that one comment is wrong, the timing of sustain=0xFF and loop=0xEB means we play for 10ms * 255 * (0xEB+1) = 601800 ms
which is around 10 minutes, not 60 minutes. I suspect that you would see xpadneo_ff_worker sending a rumble=0 command to the controller but the controller isn't following it.
I am very new to device development. haha.
Then: welcome! :heart: Let me explain some things to get you started. I'm always happy about new contributors willing to debug or add fixes/features.
The code path is like this: game rumble commands are passed to evdev, evdev puts the command into the ff-memless worker, this runs every 50ms to combine all current effects, calculating sine waves, square waves etc. At each interval, ff-memless will call into xpadneo_ff_play (which you currently log) to send us the current parameters. xpadneo_ff_play then buffers exactly one command (overwriting the previous) and schedules it to be played now, or at most 10ms from now (depending on the last execution timestamp, that's the throttling, the buffer overwrite also ensure that we don't lose the last command which may be a rumble stop command). The actual hardware work is done by xpadneo_ff_worker which prepares a hardware buffer, and then hid_hw_output_report()
puts the rumble command on the Bluetooth HID channel. Take note that the worker lags behind one jiffy (which is one timer tick of the kernel, with a 1000 HZ kernel, this is 1ms, with a 100 HZ kernel, this is 10ms, it's how worker scheduling works in the kernel). To reduce the pressure on the Bluetooth protocol, we won't send repeating identical rumble commands but instead rely on the command running for 10 minutes if we don't reprogram it otherwise. Actually, xpadneo_ff_play won't see the original timing values from the games as those are swallowed by the ff-memless emulation, so we statically go with 10 minutes (which is what the Windows driver also seems to send to the device according to packet dumps).
All this throttling and scheduling and radio usage optimization makes the code look quite complicated but it's actually a very simple worker/producer dependency, remove the complicated, deprecated, and non-working directional code and it will look a lot more clean already.
It looks like the rumble is being turned off and back on again several time in the span of less than a second...
This probably means that ff-memless emulates a square, sine, or sawtooth wave. You could graph the values into a spreadsheet by logging (jiffies,(weak,strong)) where "jiffies" would be on the time axis. Direction and duration/loop are useless if we are called from ff-memless (always 0), and we can only work on effect 0 if called from ff-memless, all others are filtered out, we'd ignore them, natively the controller does support square wave and monotonic rumble, nothing else, ff-memless emulates all the effects by updating the rumble motors at 20 Hz time resolution. I did that graphing once to confirm that the ff-memless emulation properly calls into us and supports all effects with correct timing.
I can trigger the effect pretty easily even if SDL mappings are not right I think.
The problem is that SDL assumes a different raw HID packet format. I need to change that part of the code to fulfill those assumptions. It was a pretty good idea back then as it fixed a lot of mapping problems easily for xpadneo - but that obviously only works when the driver is queried only through evdev.
The idea is outlined here: https://github.com/atar-axis/xpadneo/issues/286
Some background discussion is here: https://github.com/ValveSoftware/Proton/issues/4707
There's some more discussion on that in our discord channel.
Hello everyone, I'm also experiencing this behaviour. I'm far from being capable of debugging driver level applications but I'd like to add my two cents.
During this week Steam is holding a festival with some demos of games for us to try. One of them is the demo of Ys IX, this game triggers this behaviour everytime my character is hit by a foe. The rumble should be strong and short, during somewhere around 0.5 s, but, what happens is that I keep getting the rumble effect for few seconds, it increases than decreases than increases again and so on for about 3 seconds. When I run the game with Steam Input on, it does not happen.
I had the same experience running Assassin's Creed Origins through Ubisoft Connect, on lutris, with lutris-6.4 wine.
I'm currently on Void Linux, kernel 5.12.10_1, bluez 5.59, xpadneo-0.9.1, all installed from my distro repos, ERTM disabled and I have a Xbox Series gamepad.
From the logs provided so far, it looks like you're using the Xbox Series X|S gamepad. Thanks to ko-fi, I now own such a model to investigate issues physically. We may have already found a problem that could explain your observation: https://github.com/bluez/bluez/issues/156
That issue is about input latency but it may also affect output latency, I didn't test that yet because with 100ms input latency, it hardly makes any sense to look for rumble delays in games.
Please re-open if it is still an issue.
Version of xpadneo
git (as of 05/12/2021)
Severity / Impact
Describe the bug
In the pair of games I tested (Assassins Creed 2), the controller is supposed to do quick rumbles in certain situations (falling/getting hit for example). This works correctly when wired, but when wireless the rumble is slow and drawn out and last too long.
Steps to Reproduce
Play above mentioned games with controller in wireless mode.
Expected behavior
Faster, quicker rumble
System information
uname -a
Controller and Bluetooth information
Using clean driver, rtms should now be enabled (I can connect reliably now it seems!) Default bluetooth and module settings Intel onboard BT Controller version 1914
Additional context