[Request] Support for mouse input correction on Unreal Engine 5

[KonstantinTomashevich]

Hi! I'm participating in shooter development on Unreal Engine 5 and we're using RDP for secure remote work.

Our problem

Due to how RDP works, even if application on server grabs and hides mouse, its movement is still locked by screen borders. For example, you cannot rotate character fully, as virtual mouse collides with screen border and stops moving. The same goes for editor: when you rotate camera in editor, you cannot rotate it around as mouse gets stuck on screen border.

You can observe this behavior in almost any game when using RDP: for example, you can check Xonotic, as it is free and open source.

Steps to reproduce it in Xonotic:

• Connect to Windows desktop.
• Download example game Xonotic: https://xonotic.org/download/.
• Unpack and run game (xonotic.exe).
• Start singleplayer, first level (just for example).
• Press Join.
• Move mouse to the right as long as you can.
• After some time camera stops moving. In my case it was after two full rotations (~4*PI).
• Optional: Move mouse to the left as long as you can, it will also stop moving after some time.

This issue makes game testing very inconvenient as in first person games you need to actively rotate view here and there, but this issue "randomly" blocks you from rotating. Keep in mind that Xonotic is just an example of what happens, it is not related to the solution below.

Solution

Unreal Engine 5 provides custom solution for this problem in Windows application implementation, link to context line in 5.2.1 release source code (you need access to UE5 source code to open it, instruction how to get it -- here). This solution works for proprietary RDP client on Windows, but doesn't work for xfreerdp.

I've debugged xfreerdp from stable-2.0 branch myself for a bit and discovered that what happens is:

• When mouse pointer on server reaches screen border, RDP server sends pointer position update to client that moves mouse pointer back to window center.
• FreeRDP might ignore this update if grab mouse is not enabled.
• Even if grab mouse is enabled, X11 FreeRDP implementation might ignore it, because it might think for some reason that it is not focused.

So I've created hacky patch for myself to solve this issue locally as fast as possible (for branch stable-2.0):

--- a/libfreerdp/cache/pointer.c    (revision 0774ea7d4cf948a7a57b58baaf5ffcaf53b2ee84)
+++ b/libfreerdp/cache/pointer.c    (date 1699013667021)
@@ -68,8 +68,8 @@
        !pointer_position)
        return FALSE;

-   if (!context->settings->GrabMouse)
-       return TRUE;
+   //if (!context->settings->GrabMouse)
+   //  return TRUE;

    pointer = context->graphics->Pointer_Prototype;
    return IFCALLRESULT(TRUE, pointer->SetPosition, context, pointer_position->xPos,
--- a/client/X11/xf_graphics.c  (revision 0774ea7d4cf948a7a57b58baaf5ffcaf53b2ee84)
+++ b/client/X11/xf_graphics.c  (date 1699013667035)
@@ -560,8 +560,8 @@
    }

    WLog_DBG(TAG, "%s: %" PRIu32 "x%" PRIu32, __func__, x, y);
-   if (!xfc->focused)
-       return TRUE;
+   //if (!xfc->focused)
+   //  return TRUE;

    xf_adjust_coordinates_to_screen(xfc, &x, &y);

But, of course, just disabling this checks is a temporary hack and it needs to be resolved on other level.

How to test it

The shortest path to check whether Unreal Engine 5 solution is working for you is:

1. Install Unrean Engine 5.2.1 from Epic Games Launcher.
2. Run editor and create any non-blank game project (non-blank projects have default levels and default level is what you need).
3. Try to rotate level viewport camera (right mouse button).
4. If you can move mouse to the right for a long time while holding right mouse button and camera will continue to rotate, then solution works. Otherwise, camera will to stop rotate after some time (when virtual pointer collides with screen border).

WARNING: Do not test this in the game as it needs to be manually enabled in games and it is not enabled in official examples. The only thing where this solution is enabled by default is level viewport rotation tool.

[akallabeth]

sounds like a problematic thing to do (grab mouse without a way to release it from xfreerdp) so will most likely take some time (and testing)

but for your use case, did you look into relative mouse movement? the latest [MS-RDPBCGR] contains support for that

[KonstantinTomashevich]

I've tried this option, but RDP servers on our machines do not support it, unfortunately.

[KonstantinTomashevich]

sounds like a problematic thing to do (grab mouse without a way to release it from xfreerdp) so will most likely take some time (and testing)

Technically, it just forces mouse warp through, nothing more, nothing less. That's why just removing 2 ifs helped in my case. I think the main issue here is how to do this correctly from architecture point of view, because directly warping user mouse without checks sounds like a bad decision.

[akallabeth]

yes, just wrap around, but currently the mouse is the (fool proof) way to release keyboard grabbing (just move out of the window) when having this there needs to be an equally easy way to release the grab (and that is to be implemented for each client)

too bad there is (currently) no support for the relative mouse move in (most) windows servers :/

[KonstantinTomashevich]

Yeah, I understand that implementing it for all platforms is not an easy task.

Anyway, fixing this now is not a priority for sure: we can work with our "hack-fix" for X11 that I've posted.

My goal was to let you know that this issue exists and it will be cool to address it in future releases, but it is more of a "backlog feature request" -- we do not expect quick fix and we're fine with our local hack.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[akallabeth]

we have implemented [MS-RDPBCGR] relative mouse events. that will be the closest we can do for this here, the rest is up to the server side supporting relative mouse events.

[KonstantinTomashevich]

Thank you for letting me know. Is it already in 3.7.0? If it is, I'll try 3.7.0 and I'll see if I can somehow update or configure our server side to support relative mouse events.