Mouse capture is unsupported

[DemonicSavage]

As seen in https://github.com/waydroid/waydroid/issues/279, mouse capture is as of yet unsupported, so Minecraft (for example) is unable to be played in multiscreen setups.

Since that last bug report is closed, I'm creating this new one.

[stsquad]

So we need to know:

[ ] - how android signals it's taking the pointer device (does it even need to)? [ ] - how to capture the pointer and keep it bound within the wayland surface we are using for the application

[stsquad]

I've going to guess for the wayland side it's this unstable API: https://wayland.app/protocols/pointer-constraints-unstable-v1

[Snivine]

I found this on the Android side, there's this function for pointer capture https://developer.android.com/training/gestures/movement#request_pointer_capture

[yxles-dev]

Are you guys talking about how the mouse pointer act just like touch input?

[stsquad]

@yxles-dev I think so - at least I think the mapping from mouse to touch is done by Android itself rather than the app taking direct control of an external mouse. A common use case is Minecraft with a mouse on Android. We have fixed one bug (#279) where the mappings were reversed but on a multi-monitor screen if you turn too far in one direction the mouse cursor slides off the application window and you loose the ability to control the player.

[yxles-dev]

@yxles-dev I think so - at least I think the mapping from mouse to touch is done by Android itself rather than the app taking direct control of an external mouse. A common use case is Minecraft with a mouse on Android. We have fixed one bug (#279) where the mappings were reversed but on a multi-monitor screen if you turn too far in one direction the mouse cursor slides off the application window and you loose the ability to control the player.

Oh yeah I also experienced this even on single monitor screen. The cursor will go outside the Waydroid windows if you move the mouse too far

[TauPan]

Oh yeah I also experienced this even on single monitor screen. The cursor will go outside the Waydroid windows if you move the mouse too far

This seems to imply that if you use waydroid in full screen on a single monitor would be a workaround for this, but I've just tried with a full screen setup according to https://docs.waydro.id/faq/setting-up-waydroid-only-sessions and the waydroid ui was the only thing displayed, no window borders, panel or anything but when I moved my mouse to the far left or the far right, minecraft would stop turning around.

Are there any things I could help test?

[eris667]

Same problem as identified above, I can help test on Nobara 36 (Fedora base) specifically I was attempting to get Minecraft Bedrock/PE functional in Linux

[stsquad]

I've tracked down the pointer capture in inputflinger:

void CursorInputMapper::configure(nsecs_t when,                                                                                                                              
        const InputReaderConfiguration* config, uint32_t changes) {                   

where there is logic to deal with changing the setting:

    if ((!changes && config->pointerCapture)                                                                                                                                 
            || (changes & InputReaderConfiguration::CHANGE_POINTER_CAPTURE)) {                                                                                               
        if (config->pointerCapture) {                                                                                                                                        
            if (mParameters.mode == Parameters::MODE_POINTER) {                                                                                                              
                mParameters.mode = Parameters::MODE_POINTER_RELATIVE;                                                                                                        
                mSource = AINPUT_SOURCE_MOUSE_RELATIVE;                                                                                                                      
                // Keep PointerController around in order to preserve the pointer position.                                                                                  
                mPointerController->fade(PointerControllerInterface::TRANSITION_IMMEDIATE);                                                                                  
            } else {                                                                                                                                                         
                ALOGE("Cannot request pointer capture, device is not in MODE_POINTER");                                                                                      
            }                                                                                                                                                                
        } else {                                                                                                                                                             
            if (mParameters.mode == Parameters::MODE_POINTER_RELATIVE) {                                                                                                     
                mParameters.mode = Parameters::MODE_POINTER;                                                                                                                 
                mSource = AINPUT_SOURCE_MOUSE;                                                                                                                               
            } else {                                                                                                                                                         
                ALOGE("Cannot release pointer capture, device is not in MODE_POINTER_RELATIVE");                                                                             
            }                                                                                                                                                                
        }                                                                                                                                                                    
        bumpGeneration();                                                                                                                                                    
        if (changes) {                                                                                                                                                       
            getDevice()->notifyReset(when);                                                                                                                                  
        }                                                                                                                                                                    
    }                                     

but I can't see where the config is exposed at the low level of waydroid-hwc.

[stsquad]

So as I understand the Android input stack the InputReader is responsible for feeding the data from the underlying system so it stands to reason when the .mode is updated to MOUSE_POINTER_RELATIVE the InputReader should be able to communicate this to the underlying HW (in this case wayland_hwc) somehow.

[stsquad]

The commit 78f97b3263053c388080a738b56499139517c3b6 added support to the Input Reader for pointer capture. Maybe I can see where the underlying config happens there.

[TauPan]

FWIW I've bought an external mouse and keyboard for my android tablet where I play minecraft and it works perfectly, so in principal android supports this. (Don't know if anybody else had doubts about this.) (And that way I get all the handy bedrock mouse shortcuts on android.) (There's some funky stuff happening if I use both mouse/keyboard and touch. Focus handling in minecraft dialogs breaks in a way that the scroll wheel doesn't work any more. But that could be a minecraft bug.)

[Quackdoc]

this isn't something I plan on working on, but for any one interested in doing it, there are a few ways of going about this, however I am not sure if any of them are particularly good.

the first solution that we could support would be to capture the cursor then translate those commands to the virtual evdev device that we have. this is probably the easiest solution to accomplish.

the other solution for this would be to implement https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/110. (ill probably make a ticket for this itself). this could very well be needed to get proper support for handling controllers and other misc input devices anyways.

since we currently need to plug in the device to generate the connection, this is bad for multiple reasons, anyone who has an external keyboard and mouse will have experienced this issue if they plugged and unplugged it when waydroid has been running, you will get double input when the window is active, and you will get single input even when it isn't.

if you wanted a short term hack to get this to work, one could probably unplug their mouse and plug it back in. this would get waydroid to pickup the mouse, then tell the compositor to ignore the device, this would likely be on a per compositor basis, but sway might be able to do this with their swaymsg input <identifier> events disabled command.

no idea, this is completely untested.

[dirtslayer]

unplugging the mouse and re-plugging it in works ;)

[aleasto]

We can replicate the exact behaviour of View.requestPointerCapture with: https://wayland.app/protocols/pointer-constraints-unstable-v1#zwp_pointer_constraints_v1:request:lock_pointer https://wayland.app/protocols/relative-pointer-unstable-v1

[aleasto]

I've completed the implementation of pointer capture. Happy Minecrafting.

[stsquad]

@aleasto do you know when it hits the images? It looks like the last system image rolled around last week as well (lineage-18.1-20230408-GAPPS-waydroid_x86_64-system.zip).

[aleasto]

@aleasto do you know when it hits the images? It looks like the last system image rolled around last week as well (lineage-18.1-20230408-GAPPS-waydroid_x86_64-system.zip).

It also needs a new vendor image, which unfortunately failed to build for unrelated reasons. So next build batch, that is tomorrow.

[FoxXtreme]

I dont know about minecraft but roblox still has this issue I am using the 20230415 image

[aleasto]

The game must request pointer capturing via View.requestPointerCapture() when it is wanted, for example in game and not in the menus. Minecraft does so.

If not many games request pointer capturing we may have to setup a user hotkey to manually toggle it.

[FoxXtreme]

how would I check if its requesting View.requestPointerCapture()

[enderprism]

If not many games request pointer capturing we may have to setup a user hotkey to manually toggle it.

Is there a way to do it currently? Since Roblox disabled Wine in the 32bit client, the only way to play the game is through Waydroid. Roblox apparently doesn't request pointer capturing and my mouse movements stop at the edges of my screen. I don't own a controller so I can't fix it that way.

[eris667]

@aleasto do you know when it hits the images? It looks like the last system image rolled around last week as well (lineage-18.1-20230408-GAPPS-waydroid_x86_64-system.zip).

Tested Minecraft (Bedrock/Android) today on latest version of WayDroid and everything seems to be working flawlessly with my multimonitor setup and mouse capture!

Reference: (About - Build Number) lineage_waydroid_x86_64-userdebug 11 RQ3A.211001.001 32 test-keys

[DaanVervacke]

I've completed the implementation of pointer capture. Happy Minecrafting.

Thanks for your efforts! Would you be able to implement pointer capture for Minecraft Education as well?

[aleasto]

I've completed the implementation of pointer capture. Happy Minecrafting.

Thanks for your efforts! Would you be able to implement pointer capture for Minecraft Education as well?

The app has to request pointer capture via the android API, then Waydroid will convert it to Wayland pointer capture. It's not something we do per-app.

[RokeJulianLockhart]

https://github.com/waydroid/waydroid/issues/482#issuecomment-1701665715

https://play.google.com/store/apps/details?id=com.mojang.minecraftpe&hl=en_GB doesn't work well with a mouse and keyboard. It moves as if I've enabled mouse acceleration, and with noticeable jutter:

Screencast_20240813_211417.webm

Is this an example of it thinking that my mouse is a touch device (in which case, it would be a regression of this) or another bug, does anyone expect?