Mouse Grab not enabled for Android

[VA1DER]

Description

When Enabling the Hotkey for Mouse Grab on Android (tested on versions 6, 10, and 12), It does not activate and leaves the mouse pointer freely move in the background. This causes problems for most games when the system mouse pointer leaves the bounds of the game and, for example, activates the top Android tray. When this happens, any mouse click takes you out of the game and into the Android pull-down notifications menu.

Attempting to activate Grab Mouse does nothing. Attempting to activate Game Focus Mode (which is supposed to also grab the mouse) will activate the mode, but the mouse is not grabbed.

Expected behavior

Mouse Pointer locks to the game screen allowing for full use of mouse or trackball without activating the OS functions.

Actual behavior

System and game mouse pointers remain separate and the system mouse pointer can leave the game screen causing issues with the OS.

Steps to reproduce the bug

1. Bind Grab Mouse to physical key.
2. Start game
3. Press bound Grab Mouse key

Environment information

• OS: Android versions 6, 10 and 12 (likely all versions)

Note

Android has supported mouse grab since version 8 and other open source emulators (notably Magic DosBox) use it to good effect.

[MaksymKammerer]

Curious enough - in Uae4ARM project user @pelya have solved the issue of mouse pointer in Android (tested on Android 11) by completly removing it from this version of emulator: https://github.com/pelya/uae4arm/releases/tag/1.0.3.7

Since android mouse pointer is useless in GUI - maybe similar approach could be used here?

[glenviewjeff]

Here's more information for anyone wanting to take this on. The code that @MaksymKammerer linked to uses the SDL library SDL_ANDROID_SetSystemMousePointerVisible() function. The function doesn't appear in the SDL wiki; maybe this is a defunct function from SDL v1.

Either way, I believe to use to hide the mouse pointer natively, either use onCapturedPointerEvent(), requestPointerCapture(), or if for some reason those don't work, use an empty bitmap for a custom mouse pointer.

[InfamousStarFox]

Here is the android documentation for pointer capture: https://developer.android.com/develop/ui/views/touch-and-input/gestures/movement#pointer-capture

Once the request to capture the pointer is successful, Android calls onPointerCaptureChange(true). The system delivers the mouse events to the focused view [...] Android delivers pointer events from sources other than the mouse normally, but the mouse pointer is not visible anymore.

So, this should be possible to do natively on android since version 8 without the need to use the SDL library.

[rrwholloway]

Same issue. Especially now Dosbox Pure core works so well, not having mouse grab makes a lot of Dos games unplayable because a mouse can't be used on Android.

[Devedander]

Running into this issue also rendering a trackball useless as mouse cursor is always on screen and limits movement. Any update on bringing this in?

[worleydl]

I took a stab at fixing this. Requesting the pointer grab has a side effect of losing mouse in the UI for now unless someone takes the time to build an "internal" mouse, or make it so the mouse capture is released when the UI is shown. Unfortunately my only test mouse on my android device is a Dualshock4, because of this my patch also disables the legacy touchpad->analog mapping, I'd like to make that configurable but I'm not sure how. On the upside this let's you use a mouse/touchpad in dosbox without the aforementioned issues.

Feel free to reference this branch and pluck anything that's of use: https://github.com/worleydl/RetroArch/tree/feature/android_mouse_capture

[schellingb]

@worleydl I also gave this a shot in my android_grab_mouse branch with a CI debug build for Android available as APK here (44.9 MB). Installing a debug APK might ask to uninstall an existing RetroArch app, if doing so, make sure to have RetroArch config and data backed up or be willing to lose it.

I was close to opening a pull request with this but I feel like it's still a bit finicky. For me I need to enable the Input setting "Automatic Mouse Grab" (and restart RetroArch) to make it work. Setting "Auto Enable 'Game Focus' Mode" also works. But I think it should just work automatically. Android with a Mouse attached should behave like running RetroArch on a desktop computer in full screen.

Also you wrote

Requesting the pointer grab has a side effect of losing mouse in the UI

But does the mouse in the UI even work for you? Even without my PR in the current version of RetroArch the mouse cursor is visible on Android over the menu but it doesn't do anything for me. Setting the User Interface setting "Mouse Support" does make the interface somehow react to mouse clicks but it seems to detect a click in the wrong location and just makes the menu jump around weirdly. Is that not the case for you?

Anybody else is welcome to try it out and write back if and how it works for you, thanks!

[PatrickStankard]

@schellingb Any chance you could generate a 32bit APK from your branch? I'm looking to test this on an Arcade1Up cabinet that runs Android 10

[schellingb]

@schellingb Any chance you could generate a 32bit APK from your branch? I'm looking to test this on an Arcade1Up cabinet that runs Android 10

I made it do a new build because the old one expired and now there's a new artifact (44.9 MB). It contains normal/debug/phoenix-normal-debug.apk which contains all 4 Android platforms: 64 bit ARM (arm64-v8a), 32 bit ARM (armeabi-v7a), 64 bit x86 (x86_64) and 32 bit x86 (x86). Can you use this to test?

Again reminder to everyone that installing a debug APK might ask to uninstall an existing RetroArch app, if doing so, make sure to have RetroArch config and data backed up or be willing to lose it.

[PatrickStankard]

@schellingb this build worked perfectly for me, after I enabled "Automatic Mouse Grab", and turned on "Game Focus" mode. Thank you so much!

I had previously hid the pointer globally by replacing all of the pointer images in framework-res.apk with an empty PNG. Even with doing that, there were still problems with mouse input in RetroArch, where the mouse pointer would hit the bounds of the screen behind the scenes and get stuck. That doesn't happen anymore in your build now that the mouse grab works correctly.

Edit: I noticed that setting “Game Focus” ON before launching a game didn’t work, but toggling it on from the quick menu worked.

[PatrickStankard]

@schellingb after testing your branch build more, I think this commit might be the reason why the behavior is inconsistent: https://github.com/libretro/RetroArch/commit/23da41cd21264ae1fb6cb71476c9dbd319cc68de

Since the Android app has the option to enable/disable mouse grab on launch, I think the code there to always grab the mouse for platforms without windowed video is conflicting with that. I wonder if reverting that commit would fix it?

[schellingb]

I think if this were revisited, the goal should be:

• Android with a Mouse attached should behave like running RetroArch on a desktop computer in full screen (no fiddling with "Automatic Mouse Grab" required)
• The UI should be controllable with the mouse just like on a desktop computer

Now that I think about this, I wonder how mouse support works in Linux with DRM/KMS. Retroarch runs via DRM where it forgoes the X Window system altogether. Is there mouse support? Does it work like on a desktop? I have a Raspberry Pi where I could try that out some time. If DRM works better, hopefully Android with a mouse could be made just as good. Or maybe it is also broken :-)

[i30817]

This problem also existed in Wayland but there was a recent series of commits¹ that fixed it. One of the problems ensuing from the fix until it was reported and fixed was with the settings -> input -> auto enable game focus mode crashed ra.

This crashed ra because it tried to lock the cursor when it was already locked by the core (iirc, automatic for dosbox). Something to test on your commits.

¹ https://github.com/libretro/RetroArch/pull/15103 merged march 20 https://github.com/libretro/RetroArch/pull/15114 merged march 21 https://github.com/libretro/RetroArch/pull/15128 merged march 24

This bug was also the cause of cursor movement and acquisition bugs in scummvm, nds and 3ds cores, quake and dosbox cores, among others probably. It needs solving.

Android default UI doesn't actually use the mouse for anything, but the cores need to be able to set it captured and centered (and most often invisible), otherwise many many bugs happen.

[i30817]

@vanfanel @ColinKinloch , could you take a look?

[PatrickStankard]

I think this commit from @schellingb 's branch on it's own fixes the bug: https://github.com/libretro/RetroArch/commit/ff07bd4f10339d8debd2ba299bcf9fb8a9613e8a. If it was cherry-picked, I think that's all that would be needed to close out this ticket

[i30817]

If it requires the mouse grab setting to on, to work on cores that use mouse grab, i disagree it's enough.

Games should work regardless of frontend setting in this case.

Besides, RetroArch android UI doesn't even use the mouse, I doubt most people could make the association 'dosbox\scummvm mouse working weird => enable RA mouse grab'.

I'm also curious if the problem occurs with overlay 'mouse'. I don't have a game with free\rotoscoped camera to test in scummvm\dosbox right now.

[dannymosquito]

Thankyou so much for everyone that contributed to this. This was driving me nuts. Tested the aarch64 apk from schellingb's link above and it works great! I'm only new to this so not sure how these branches end up getting committed to the main Android build but for my two cents even though I agree that it would be nice to have it automatically just enable the "mouse grab" option however at the end of the day it would have saved me a lot of headaches if at the very least this current change was part of the main Android version. Selecting the "mouse grab option" and "auto focus" was like the third thing I tried when originally trying to get this to work and much to my dismay didn't fix it... Then spent an age trying so many other different options and workarounds all coming up with nothing. At the end of the day this fix at least gives people a solution to the issue as opposed to the current build that in my opinion is almost unusable when using a physical mouse with no real workarounds. Just my thoughts. Thanks again...

[Freegavin27]

Can anyone help me get an APK for @schellingb 's android_grab_mouse branch? The link appears to be dead now.

[dannymosquito]

Can anyone help me get an APK for @schellingb 's android_grab_mouse branch? The link appears to be dead now.

Hi Gavin... I can't attach it here because it says it's too big. How else can I get it to you?

[PatrickStankard]

I ended up forking @schellingb's RetroArch repo, rebased the android_grab_mouse branch off of the latest libretro/RetroArch:master, and reverted this commit that was attempting to auto-grab the mouse for platforms without a windowed mode.

This is what the diff for the android_grab_mouse branch looks like with those updates: https://github.com/libretro/RetroArch/compare/master...PatrickStankard:RetroArch:android_grab_mouse

I tested this on my Android cabinet and it works perfectly now - the only thing you need to do is enable "Automatic Mouse Grab" in the RetroArch input settings, restart the application, and it's all set going forward. The pointer is hidden as expected when using the mouse for input. It's worth noting that if you don't do this, the mouse input will still work, but you'll just see the pointer over the RetroArch window.

Here is the artifact (46 MB) that has the build for the updated android_grab_mouse branch, the APK is under normal/debug/phoenix-normal-debug.apk.

I really hope this change makes it in! I know a lot of people running RetroArch on Android will be thrilled to see it (myself included). Thank you for all of your work on this, @schellingb 🙌

[PatrickStankard]

This should be fixed now that #16203 was merged, if anybody wants to test it on the latest nightly