Open waheedullahkhan001 opened 2 months ago
rom1v
commented
2 months ago Not about acceleration, but pointer speed:
Does it help?
Scrcpy only sends mouse HID events, how the device handles them is out of control of scrcpy.
waheedullahkhan001
commented
2 months ago The problem is mouse acceleration. Many people like to disable mouse acceleration as it helps with muscle memory and afterwards using mouse acceleration feels really weird.
Maybe there is a way to intercept those HID event and apply the deceleration factor? Might be too much to ask for but, it would be a really cool feature for those who want to try gamming using this. 😅
anotheruserofgithub
commented
2 months ago Following https://github.com/Genymobile/scrcpy/pull/4473#issuecomment-1965152264: I encountered the same issue when I implemented uhid/uinput injection in JNI (to make the mouse cursor appear on the device's screen). My problem was that the mouse on the device did not display the same trajectory as the mouse on my computer, because Android actually applies acceleration to the mouse input events.
This issue could be easily highlighted if you were able to draw on screen the motion that is expected by each mouse event (meaning, the injected increments) and in parallel observe the trajectory of the mouse pointer. The mouse trajectory will shift drastically.
It turns out that if you plug your physical mouse directly to the device, then this acceleration would typically disappear (at least with my mouse) and I assume this is because mice usually are hidraw devices, or because mouse vendors cancel out this effect in their drivers, though I couldn't find any specific public code anywhere (nor in Linux kernel drivers).
Even if you accepted this acceleration as a built-in Android feature, there is a problematic effect remaining. If you analyze the mouse motion closely (when it's captured by SDL, perform slow linear movements), sometimes you can see what look like bumps that are not due to the communication latency itself but to the acceleration being applied to injected events, which arrive on the device with uneven delay compared to when they were generated on the computer. Hence the mouse velocity as computed by Android can become very much wrong because it's based on noisy/wrong timestamps, leading to unexpected "jumps" or (on the contrary) "brakes". Those should be smoothed out if possible, and I will shortly explain how it's possible.
A typical uhid run with my device (I moved the phyiscal mouse at almost constant speed):
The time interval between two mouse motion events is typically 10 ms but sometimes it's only 1 ms.
At first, I tried to reverse-engineer this acceleration in order to cancel it out, but I couldn't achieve any satisfying result. However I found out a simple solution to work around this issue: inject events only every 40 ms and accumulate (merge) increments in the meantime in order to not block the process. That works because the history of the velocity tracker is cleared after 40 ms without events (see also the header file), so if you inject one event/sample after 40 ms, the tracker won't be able to compute a velocity with a single sample, thus no acceleration will be applied. It will just send the raw increments, which is what we want.
Although there have been some recent changes, if you check the history you will see that this behavior has remained quite consistent since the beginning of Android. Lately it looks like they changed a >= condition to > and added a velocity tracker for the wheel (which means we might need to cancel acceleration also on scrolling), but apart from that I don't see anything that might prevent this solution from continuing to work.
I can share some C code but this part is pretty straightforward, you just need a bit of synchronization with a background thread that wakes up every 40 ms to do the injection when events remain to be processed, or waits for new events to come up (e.g. using a condition variable, but in Java you might simply use Object.wait() and Object.notify()).
In my opinion, that's the cleanest and most generic solution. Of course, this means that you will only get (at best) a 25 Hz cursor refresh rate, but for normal usage this will be good enough (maybe not for gaming? see https://github.com/Genymobile/scrcpy/issues/269 and https://github.com/Genymobile/scrcpy/labels/game but https://github.com/Genymobile/scrcpy/pull/2130 may not be impacted as it's not the same kind of device). So we could have either a boolean option to enable/disable this workaround, or an integer to specify the accumulation time:
@rom1v What are your thoughts on these aspects? Sorry for the lengthy message.
rom1v
commented
2 months ago Thank you for your investigations and details!
Mouse event resampling could also solve a separate problem: #3088. So it might be interesting to add an option --mouse-resample=40 for 40ms. For #3088, it seems perfectly reasonable to perform resampling on the client side (where the events are generated).
However, there is one difficulty for mouse compensation.
This option should work for all mouse input modes (sdk, aoa, uhid). In AOA mode, the events are injected the client side, while in SDK and UHID mode they are injected on the device side.
For more precision, the resampling should be applied as close as possible as event injection: if we resample to send an event every 40ms on the client side, we might end up with events separated by 37ms or 43ms on the server side due to network jitter, especially if scrcpy is connected wirelessly. This would break the assumptions about mouse acceleration behavior.
But on the other hand, it is annoying to perform this mechanism both on the client and on the server (depending on the mouse input mode). Worse, for UHID, HID data is generated on the client side (so that the same code can be used for both AOA and UHID). Resampling should occur before serializing HID data, so it could not be performed on the server side (where it does not "understand" HID, and it would not be satisfying to deserialize HID, resample and reserialize HID).
So maybe we should resample on the client side, and "hope for the best", even if sometimes events might be separated by less than 40ms on the server side. It's an heuristic anyway (the implementation in Android could change in theory).
anotheruserofgithub
commented
2 months ago I totally agree with your concerns. For sure I can tell that resampling on the server side every 40 ms produces pretty smooth movements, even wirelessly through a VPN as far as I remember, and with reasonable delay with respect to the computer. But I'm afraid that resampling on the client side would still induce jittery movements if the server receives events at only 25 Hz on average. As you said, sometimes there can be a non-negligible difference in time intervals, and perhaps that would be noticeable at such a low refresh rate, but I don't know. And this may "fail" if events get injected below 40 ms, but that should be negligible since the mouse is captured by SDL so at least we won't see two drifting pointers on screen.
anotheruserofgithub
commented
4 weeks ago Follow-up of https://github.com/Genymobile/scrcpy/pull/4473#issuecomment-1965152264 & https://github.com/Genymobile/scrcpy/issues/4697#issuecomment-1965578037.
@rom1v I took the time to clean up and rebase my old code on latest release (it was originally developed upon v1.20). Please find it in this zip: I don't want to maintain a fork open, but feel free to add it in a branch of your repository if you like.
Also, I dug into the Android source code to find references. It turns out I have some good news to share, at least I hope so.
My code is mostly a native library (with JNI) to inject relative mouse motion events. It provides the ability to highlight the mouse acceleration applied by Android, and to demonstrate how suitably resampling events allows to cancel out this acceleration.
For information:
History of VelocityTracker.cpp:
frameworks/base/libs/androidfw to frameworks/native/libs/input>= condition to >Where acceleration is configured and applied:
By default, if you run scrcpy, mouse motion events will also be injected as UHID events in the native library so as to show the cursor on the device screen.
In the last commit I disabled the 40ms waiting time between events, so you will see acceleration being applied to the mouse by Android.
The current mouse position is received in absolute coordinates, and a calibration procedure is applied on the very first event, by forcing the mouse to a corner of the screen, where its position is capped by Android. Then the last mouse position is stored in order to be able to further inject relative movements.
When running scrcpy -M, i.e., the built-in UHID mode, I changed the server side with a dirty hack to forward motion events to the native library instead.
Hence this will allow to show the difference when acceleration is compensated with resampling.
But first, with the last commit, acceleration is applied as usual. You can see that by following these steps:
scrcpy -M,Revert the last commit in order to enable the 40ms resampling that cancels out acceleration.
Repeat the experiments with both scrcpy and scrcpy -M, and you will observe two beneficial effects:
The cost is obviously a slightly increased delay between the cursor moving on the computer screen and the pointer icon chasing it on the device screen.
Recently, Android added an API to set the pointer acceleration directly.
Pass 1f to achieve the same result (I guess) as event resampling (except for smoothing out communication jitter).
Maybe users would like to have an option to tune acceleration?
Also, it seems that they provided APIs to create virtual devices through UINPUT.
See the links below.
Pointer acceleration:
Built-in virtual UINPUT devices:
To sum up:
Is your feature request related to a problem? Please describe. When trying to game using
--otgI realized the presence of that annoying mouse acceleration. This tool could be really useful for gaming (or maybe even normal use can be improved) only if it had this feature and key mapping (but maybe there are tools for that already?).Describe the solution you'd like A flag where we can pass a value which will determine how much deceleration (or whatever is the opposite of mouse acceleration) to apply before passing the mouse input. Better yet hardcode a value which will always work (if that is possible).
Describe alternatives you've considered I tried searching the web and in my phone but I couldn't find a way to disable mouse acceleration without rooting.
Additional context Not needed?