ReenigneArcher
commented
1 month ago @ABeltramo do you have any thoughts on this?
Closed alecfriedman3 closed 3 days ago
ReenigneArcher
commented
1 month ago @ABeltramo do you have any thoughts on this?
ABeltramo
commented
1 month ago Admittedly, I've only tested and played with Moonlight QT. My first suggestion would be to test if this can be reproduced on another Moonlight client when using the same Joypad, just to exclude some issue with Android.
When using "gyro as joystick," the problem is masked and gyro is still mostly usable (the cursor exhibits small choppy jumps of a few mm on screen), but when using "gyro as mouse" it's incredibly pronounced and completely unplayable (cursor flies across the screen several inches at a time).
Are those options that you set in Moonlight or with some re-mapper tool on the host?
alecfriedman3
commented
1 month ago I could probably test with Moonlight QT with an actual dualsense, though not with the gamepad (the gamepad is integrated into the android device). I could also give you the log of raw android gyroscope sensor values if that's helpful.
Should have been more clear with this one. Gyro as joystick (and mouse) are settings in Steam, so "gyro as joystick" is really just masking the problem since input is translated to analog stick output, and then passed to games which apply their own smoothing and acceleration to analog stick output. Gyro as mouse actually translates directly to mouse dpi, which makes it incredibly precise, but also much more sensitive to signal noise and less fault tolerant to disruptions.
Basically if you think about how values like this might be processed in either case, with a quick bump of an analog stick returning to zero, the movement of an in-game crosshair would be bounded by the max cursor speed and smoothed by acceleration (and in this case the bump is "instantaneous" from near 0 to -1677721, so there's no actual analog travel; the game sees the stick as pushed all the way over "moving at max speed" for an instant, and then immediately back at baseline). But quickly sliding a mouse and coming to rest will always directly move the cursor relative to your mouse's dpi setting, so steam sees these gyro values and applies them to your game as if your mouse just instantly moved -1677721 units of output and then stopped.
I can try to take some screen recordings as well to show what the difference between these two behaviors looks like. They're both very distinct observable behaviors, that line up with how those two modalities should behave when translating bad values to mouse and joystick output respectively.
ABeltramo
commented
1 month ago Thanks for clarifying, I get what you mean I just wanted to try and replicate it so that I could see where the error actually is (or if that's just something that gets wrongly sent from the client). Maybe the raw readings from Android would highlight something that could be helpful.
Either way, I think it's sensible to put some better checks for outliers if we don't already. I'm in the middle of a rather large PR in Wolf at the moment, but I can take a better look at it in the next few weeks; feel free to ping me if this doesn't get any attention from anyone else.
alecfriedman3
commented
1 month ago Yeah sure thing, thanks. This is gyroscope raw sensor data collected over ~1 minute, Gyroscope.csv. Should be in radians/second according to Android's docs, which does go through a radian-deg-radian round trip at some point while it travels through moonlight and sunshine. Though I did roughly confirm the round trip calculations appeared "correct" by adding some extra logging to sunshine.
The bad value that specifically causes those -1677721 readings is this: -0.0004124999977648258. I tried replacing those values with a crude hack conditional, subbing both 0 and the next largest value that appears in that dataset for it, but both still produced choppy (though definitely less choppy) results, with that value entirely gone from evtest's output (so that's almost definitely a "bad value").
This kind of makes sense since replacing real rotation for a synthesized value would artificially produce more or less output than the value itself would, which wouldn't align with your actual movements. However the choppiness observed in this case also matched pretty closely to the choppiness of the real dualsense through android (which never produced those "bad values" in the first place), which makes me think there's possibly still smoothing that should happen somewhere past a simple exclusion filter.
I'll give a test on moonlight qt too this week and report back. That section of the Android docs do also have this note below. So perhaps if moonlight-qt is totally smooth, then the proper fix really would belong in moonlight-android
Typically the output of the gyroscope is integrated over time to calculate a rotation describing the change of angles over the time step...
...
In practice, the gyroscope noise and offset will introduce some errors which need to be compensated for. This is usually done using the information from other sensors, but is beyond the scope of this document. Thanks, this is definitely helpful. Which specific device are you using, btw?
I feel that this might be some weird issue client side, we can probably mitigate the outliers server-side, but I wonder if this should be better fixed client-side maybe with a lower frequency for the reports depending on some parameter of the gyroscope? It would definitely help if you can use a bluetooth DualSense, at least I could try to replicate that on my phone.
alecfriedman3
commented
1 month ago I'm using an Abxylute handheld, but I also tested on a family member's android device (which I believe is a Galaxy S20 Ultra though I'm not 100% positive of that). Both android devices consistently showed the -1677721 value popping up in evtest while holding steady.
I also should clarify I did connect an official dualsense over bluetooth to the Abxylute, and while those specific bad values were not present under that circumstance using moonlight-android, the gyro movement as mouse was still choppy/jumpy in comparison to when the dualsense was connected directly to the host (also over bluetooth). It was just significantly less choppy than when those bad values were present using the abxylute's internal gyro readings. So this may be technically two separate issues; choppy gyro and bad value conversion; that could potentially be solved with one fix.
When I get the chance I'll try to screen grab some gameplay under each of these circumstances:
(dualsense over bluetooth to abxylute, with gyro as joystick, will likely appear normal, so I didn't include it)
I do wonder if it could partly be a weird client side issue though like you said. Moonlight-android apparently filters duplicate readings and does not send anything, which could in theory lead to an irregular reporting frequency back on the host. Not sure if that matters, but Moonlight-qt does not seem to do this filtering from what I can tell.
So basically if I find 4. above is smooth while 3. is choppy, then that primarily implicates the android app. And if the android app will need a fix for that anyway, it would probably be possible to just filter out the bad values we know are "too small" (causing -1677721) from the client side.
Cuc001
commented
1 month ago Im not sure if my comment would help but i have the same problem on windows 11. It started when i switched to a router with wifi 6e. Gyro as mouse is inconsistent but playable. The gyro movement is not always choppy, but when it starts the cursor flies all over the screen for a few seconds ( 2 or 3) then its back to normal. Usually i shake the phone a bit and it recovers. Its not happenning on steam remote play where i am even able to play with gyro TO mouse, which is way smoother. The client is a samsung s23 which recently got an update to enable wifi 6e in my country. Before the update, the gyro problem was worse. The router is an asus axe7800. This problem is annoying me and i hope it will be fixed some day.
alecfriedman3
commented
1 month ago So I was able to do some more testing last night. I'll upload some video links when I get a chance later, but the short answer is that the dualsense connected to the android device (3 above), and the dualsense connected to a windows moonlight-qt client (4 above), both behaved similarly. Which was choppy and inconsistent with gyro to mouse, where sweeping movements were managed alright but low-rotation movements lead to the cursor flying around (in fact I even thought dualsense over android was actually slightly better than dualsense over QT).
When using the android's gyroscope (1 above), the cursor would similarly fly around, but there was also a distinct additional "negative" pull (most of the flying was down-left-down-left), which would definitely be from those -1677721 values. This is all in contrast to playing with "gyro to mouse" over steam link, which is basically flawless like @Cuc001 said (I'll upload that video as well for comparison). @ABeltramo is inputtino used for windows hosts as well?
(I also realized "gyro to mouse" and "gyro as mouse" are different modes. I'd really been referring to "gyro to mouse" mode above, which is the one that actually converts movement to dots per inch, not as mouse. I imagine "as mouse" probably falls somewhere between "as joystick" and "to mouse" in terms of usability right now).
alecfriedman3
commented
1 month ago Okay here are the screen recordings:
1 and 2. Moonlight Android - Abxylute - Gyro to Mouse and Gyro as Joystick (later in the video)
3b. Moonlight Android - Dualsense Gyro to mouse, gameplay
alecfriedman3
commented
1 month ago Okay so big info dump here, apologize for the length but it's all pretty relevant. I think there is something to be said about the two different mouse modes in Steam. "To mouse" is a newer mode that's only been out a little over a year (though it's unclear whether Steam plans to deprecate the old as mouse mode eventually or leave them both running). The differences between the two modes are that, "to mouse" always outputs a 500hz signal, while "as mouse" does not (confirmed by using a mouse polling rate tester). Additionally, "to mouse" actually uses timestamps when interpreting the gyro data, while "as mouse" simply applies the incoming signal without timestamp checking (this is according to a blog post from when the "to mouse" mode was first released: Gyro sensor timestamps are now used resulting in better overall accuracy of the Gyro., which implies timestamps were ignored previously with "as mouse").
Sunshine only ever requests a 100hz signal for motion sensor data, and additionally moonlight (android) filters duplicates, which leads to a sampling rate of sometimes as low as 50hz on a given axis (according to me roughly calculating the frequency of evtest timestamps). But when I connect my dualsense directly to my host machine over bluetooth, and read evtest motion data timestamps, apparently it's sampling at nearly 1000hz! (I was under the impression dualsense was limited to a 250hz polling rate unless it's overclocked, which I haven't done, but I'm not sure if that data can really lie? Though I'm not one to complain about hidden performance gains...)
Anyway all this to say, after some more testing, "as mouse" with a real dualsense controller over moonlight-android seems to work pretty well. It's smooth and there are no large jumps that I've noticed at all. The android gamepad itself still suffers from the extremely small negative value jumping problem however. Though when I use the crude hack to replace those small values with the next smallest known working value, "as mouse" with the android gamepad actually does work just as well as the real dualsense. My thoughts then with "to mouse" vs. "as mouse," are that the choppiness with "to mouse" must be due to some combination of signal upsampling and timestamp interpretation that Steam is doing that doesn't agree with the signal being generated from moonlight/sunshine/inputtino. Considering "as mouse" works fine enough though with the hack, I don't particularly care if the signal is fixed to work with "to mouse" at the moment (if Steam ever officially deprecates "as mouse" I'd probably open a new issue to track that. Maybe I'll try to figure it out on my own just for fun, but not a huge priority anyhow).
So with all that, let's keep this issue focused on those small bad values that I initially opened the issue for. We can either hack the fix (since that does in fact work), or try to figure out what's going on with those values under the hood. The only thing left I could consider there, and the only other data that seems to go into driver's calculation to derive the evtest values, are the gyro calibration data values. However I tried zeroing out all the calibration data in inputtino and the rotation vectors that were causing the -1677721 value actually caused the maximum magnitude negative value for the axis to be output instead (-2097152) which I think basically rules out the calibration data as that would suggest the problem is occurring "before" it's applied anyway.
If it's ultimately too hard to track down, I'm honestly fine with a hack. I'm happy to open a PR for it if that's the way we go, but would want an opinion from official maintainers on whether that should go in Sunshine or moonlight-android. I'm inclined to believe it belongs in Sunshine though (perhaps behind a config flag even), as while this has only occurred on android that we know of so far, there's nothing to say another client couldn't output equally small vectors (gyro has not been implemented in ios yet for instance). I'm also fine continuing to build sunshine from source myself with the hack for now if you want more time to try and figure out a proper fix @ABeltramo
ABeltramo
commented
1 month ago Sorry for the very late reply, I do have a (ever growing) list of todos, and I'm definitely going to look into this at some point; I'd like to get to the bottom of it and see why those values are causing issues; possibly with a proper unit test since I've got everything already setup in inputtino.
I'd say that generally, checking for abnormal values in the backend should be the right approach instead of moving it to all clients. I'm just not sure that discarding those specific values would be the right approach here and what exactly classifies as "wrong" values. Hardcoding a few valuues just because you encountered them in the wild just doesn't feel right.. 😅
alecfriedman3
commented
3 weeks ago Yeah, I don't think that's necessarily the right fix, more so that there is a value at which this starts happening, and this is just a close point to there that doesn't exhibit the problem. In a sense clamping to 0 is equally arbitrary, but zero tends to hold more semantic meaning which makes it feel generally more acceptable (a little more on this in a sec).
But regardless, I actually think I've narrowed down the real problem (and apologize in advance because this is gonna be super long and gets pretty into the weeds of bitwise integer representations). The first thing that tipped me off was that, when I replaced this method call with an explicit type cast, everything worked perfectly. I'm assuming that only works because my host is already little endian encoded, it probably would work on a large number of hosts, but maybe that doesn't make it the best fix.
The problem is ultimately a combination of two factors though, and it only happens on very small magnitude negative values. The reason has to do with first how those values are type cast, and then when/how the signing bitwise operation is applied. htole16, on little endian encoded hosts, performs basically a simple cast to a 16 bit unsigned int. When fractional floats are cast to ints, they're rounded to the nearest int (or possibly floored/truncated, though the distinction doesn't really matter too much here). So when a value like 0.04 is cast to an unsigned int, it actually becomes 0.
So that's the first factor, which isn't inherently problematic (considering 0 actually does work in context). But it's this combined with the next part that creates those bad values. The bitwise operation on the following lines takes the decimal 1 and shifts the rightmost bit 15 places to the left which results in a bit pattern 1000 0000 0000 0000. This is then bitwise ORed with our value that was cast to an unsigned int (which is 0 in this case, or 0000 0000 0000 0000). This gives us a signed value bit pattern of 1000 0000 0000 0000, or "-0"? Except "-0" doesn't exist, and this bit pattern is actually used to represent the smallest value possible based on two's complement: -2^(n - 1) (n-1 because the 1st bit is now reserved for signing). When n (the number of bits) is 16, that gives us -32768, which is the smallest signed 16 bit int.
From here, negative values are actually encoded counting up toward 0, essentially in reverse from how their positive absolute values are encoded. In other words, -32767 is equal to -32768 + 1, and so is encoded "as if it were a 15 bit unsigned (implicitly positive) 1" with a negative signing bit that actually represents starting at -32768; or 1000 0000 0000 0001. In this way, -32768 actually is negative zero in a sense! The reason for this pattern has to do with keeping binary arithmetic consistent.
Now the problematic rotational value was -0.0004124999977648258, the gyro resolution factor multiplies that by 1145, which gives ~-0.47. The next largest value, -0.0009624999947845936, when multiplied by the gyro resolution factor produces ~-1.10. So the first value produces "-0" when cast, and the second does not. This value is then multiplied by the gyro calibration data in the underlying driver, which is why we don't actually see -32768 itself surface. Circling back to the arbitrary semantics of 0, this incidentally means zero would in fact have been a "good" choice to outright replace the value with, but I still don't really think that's correct.
It may be enough to simply add a != 0 check to that conditional, but ultimately I'm questioning whether the bitwise OR is even necessary at all actually. Negative values are cast to unsigned ints counting down from 65535 (the maximum 16 bit unsigned int value, (2^16) - 1); if you jump into a cpp playground and static_cast a float of -1 to an unsigned short int, it actually produces 65535. Negative values are cast from the top down, meeting positive values in the middle at 2^15. Considering how negative signed ints are encoded, this actually makes a good deal of sense!
What this means then is that when a value of -1 is cast to an unsigned int and becomes 65535, it's actually encoded as 1111 1111 1111 1111. This is in fact correctly the signed encoding for -1 using two's complement! And specifically -1 is not encoded as 1000 0000 0000 0001, simply flipping the sign of the positive value, because that bit pattern represents -32768 + 1. Because the cast to an unsigned int itself actually already contains the proper bit pattern to represent the values as signed ints, the bitwise OR is actually a no-op.
But it actually goes further! In theory there's an additional problem when any values wrap around the signed boundaries as well, though in practice this may be mostly a non-issue. Consider a value of -32769, cast to an unsigned short int will become 32767, which would be represented as 0111 1111 1111 1111. When this is bitwise ORed with "-0", we get 1111 1111 1111 1111, which is the signed encoding for -1! This means large enough negative values will exhibit somewhat of an inverse problem, a "stickiness." Now, a wraparound problem actually also already exists for large positive values that cross that threshold, and those values don't even receive the bitwise OR. When positive 32769 is cast to an unsigned int, it remains 32769, which is represented bitwise unsigned as 1000 0000 0000 0001. But in terms of signed ints, we know 1000 0000 0000 0000 ("-0") is actually -32768, and negative signed ints count "up from negative zero," so this value is only one greater than -32768, or -32767. Which means sufficiently large positive values actually become large negative values!
So what I think this means, is that the correct fix is removing the bitwise OR entirely, and clamping values between -32768 and 32767 inside to_le_signed, before they're cast to unsigned ints, like this. I'm happy to submit this proper fix (rather than the totally arbitrary hack job). This should all be unit testable pretty easily I imagine, though I haven't taken any time to write them yet (also happy to take a stab at testing though when I get a chance).
ABeltramo
commented
3 weeks ago Thanks for the long explanation, that makes perfect sense, and it seems to be right track; I'm glad you've found the actual underlying issue, nice job!!
I'm happy to submit this proper fix (rather than the totally arbitrary hack job). This should all be unit testable pretty easily I imagine, though I haven't taken any time to write them yet (also happy to take a stab at testing though when I get a chance).
If you want to start a PR I can help you out with setting up unit tests, bear in mind that these wouldn't work on CI because Github Actions don't expose uinput or uhid, in Inputtino and Wolf I hide this kind of tests behind a Cmake flag (and only run them locally), defined here.
To compile them you'll have to simply add -DTEST_LIBINPUT=ON, target inputtino_tests and then run the compiled binary. If you use an IDE like VSCode it should automatically pretty print the results and show the green/red checks over each test.
The way I've setup my unit tests is by reading back the values using SDL2; this way we have a 3rd party independent library that validates that what we send via the API is what's actually received back on the other side. You can see and edit the test defined here it should be fairly readable, feel free to ping me if you need any help!
ABeltramo
commented
3 weeks ago Just a side note, having a long explanation like that on Github is perfect because you can easily reference it with a little comment in code. Don't be afraid of writing down your thought process; it'll be invaluable when someone else will read thru the code especially when you're trying to wrap your head around these little details that you just found out!
ABeltramo
commented
3 weeks ago @alecfriedman3 I've opened up a PR #17 and quickly tested the changes locally (if you could check the flag that allows edits by maintainers I can also push those changes).
I've unit tested the following values:
-32769.0f becomes -28.59546f when read back by SDL32769.0f becomes 28.59546f -0.0004124999977648258f becomes 0Other non-problematic values like 0 and PI seems to work as it used to. Does that resonate with you?
I'll give it a go in-game over the next few days to get a feel of how it's working now..
alecfriedman3
commented
2 weeks ago Hah, yeah never a bad thing to have details written down. Sorry I haven't had a chance to get back to this, been pretty busy here. Thanks for making the test changes. Oddly that flag isn't present on the PR, so I just added you as a collaborator to the fork instead.
ReenigneArcher
commented
3 days ago Should be fixed by https://github.com/LizardByte/Sunshine/pull/3401
Thank you everyone involved in the fix!
alecfriedman3
commented
3 days ago Awesome!! Thanks for the help on this everyone!
Is there an existing issue for this?
Is your issue described in the documentation?
Is your issue present in the latest beta/pre-release?
This issue is present in the latest pre-release
Describe the Bug
Gyro movement on linux host is choppy to the point of being nearly unplayable. When using "gyro as joystick," the problem is masked and gyro is still mostly usable (the cursor exhibits small choppy jumps of a few mm on screen), but when using "gyro as mouse" it's incredibly pronounced and completely unplayable (cursor flies across the screen several inches at a time).
More context:
Gyro is still when controller is placed on a flat surface and not moved at all. Gyro functions mostly smoothly when moving in large sweeping motions. Gyro does not work when holding the controller "still" in your hand, which means the problem (part of it at least), has to do with very small non-zero rotational movements.
Expected Behavior
Gyro movement is smooth in all circumstances.
Additional Context
Tested on pre-release
v2024.718.23456installed with rpm. Also testedv2024.928.45639built from source.Unclear if this behavior originates from
Sunshine,moonlight-android, orinputtino, though I think fixing in sunshine may be sufficient. The reason I believe this is, moonlight-android is actually sending proper raw signals as far as I can tell, confirmed by logging sensor values from an unrelated android app which align with sensor values when logging is added to sunshine's source code. When inspecting the final gyro output fromevtest, there are a number of "bad values" present from android's raw sensors, namely-1677721, which is certainly not coincidental (while not exactly "relevant logs" as it relates to sunshine's logs, I've dropped the output in the "relevant logs" section below)Believing this were potentially the sole issue, I also connected a dualsense to my android device and tested that over moonlight. These bad values did not appear in the dualsense's
evtestoutput, however the dualsense controller still exhibited the same choppy behavior when "held still" (but not perfectly still). I believe the reason these bad values did not appear for native dualsense gyro is that dualsense is probably simply not outputting rotational values that are quite as small in magnitude as the android sensors (unfortunately I'm not able to log the dualsense raw values as read by android, only the android device's sensors, but when adding conditional logging in sunshine for values of incredibly small magnitude, nothing logs for dualsense while there are logs present when using the android device's gyro values). I also tested an additional android device which did output these bad values as well, which additionally rules out sensor problems in my android device.Interestingly these bad values only occur for negative values of incredibly small magnitude, which could implicate this code from inputtino
So it seems there are two issues that are perhaps interrelated here:
Ultimately this all leads me to believe a low-pass and high-pass filter applied to the gyro output in Sunshine would fix the issue entirely. It should naturally eliminate those bad values, as well as generally suppress noise from other high and low magnitude outliers (ie: smoothing the curve), without needing to touch inputtino. (Though I also realize it's possible the onus is entirely on inputtino, and I'm happy to open a corresponding issue there if you think that's the case)
Host Operating System
Linux
Operating System Version
OpenSUSE
Architecture
64 bit
Sunshine commit or version
v2024.718.23456
Package
other (self built)
GPU Type
AMD
GPU Model
Radeon 5700XT
GPU Driver/Mesa Version
24.1.7
Capture Method
X11 (Linux)
Config
Apps
No response
Relevant log output