Relative mouse movement

[marino-mrc]

I'm using turbovnc + virtualgl with steam games and I have a problem with mouse during the game (I tried only with half life2 and transmission).

Used version: VNCSERVER: [ntt@fedora22 bin]$ ./vncserver --help TurboVNC Server v2.0.95 (build 20160909)

CLIENT: ntt@juniper:/opt/TurboVNC/bin$ ./vncviewer-java --help TurboVNC Viewer v2.0.95 (build 20160909) [JVM: amd64]

Basically the mouse works well during the vnc session, in the steam menu and in the game menu but when I start to play it becomes "crazy" and the game is out of control. If I press "esc" the mouse works normally and I can exit the game using the half life menu.

logs of vncviewer with -loglevel 150 during the game: http://pastebin.com/dYFpU5tN out of xinput list --long : http://pastebin.com/Y6zSFKL5 log of the vncserver:

20/09/2016 09:41:19 Got connection from client X.Y.Z.K 20/09/2016 09:41:19 Using protocol version 3.8 20/09/2016 09:41:19 Enabling TightVNC protocol extensions 20/09/2016 09:41:19 Advertising Tight auth cap 'VENCRYPT' 20/09/2016 09:41:19 Advertising Tight auth cap 'VNCAUTH_' 20/09/2016 09:41:19 Advertising Tight auth cap 'ULGNAUTH' 20/09/2016 09:41:20 Client requested VeNCrypt sub-type 258 20/09/2016 09:41:20 Deferring TLS handshake 20/09/2016 09:41:23 Full-control authentication enabled for X.Y.Z.K 20/09/2016 09:41:23 Pixel format for client X.Y.Z.K: 20/09/2016 09:41:23 32 bpp, depth 24, little endian 20/09/2016 09:41:23 true colour: max r 255 g 255 b 255, shift r 16 g 8 b 0 20/09/2016 09:41:23 no translation needed 20/09/2016 09:41:23 Enabling full-color cursor updates for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Desktop Size protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Extended Desktop Size protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 rfbProcessClientNormalMessage: ignoring unknown encoding -307 (fffffecd) 20/09/2016 09:41:23 Enabling LastRect protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Continuous Updates protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Fence protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling GII extension for client X.Y.Z.K 20/09/2016 09:41:23 Using tight encoding for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG subsampling 0, Q100 for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG quality 95 for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG subsampling 0 for client X.Y.Z.K 20/09/2016 09:41:23 Using Tight compression level 1 for client X.Y.Z.K 20/09/2016 09:41:23 Using 1 thread for Tight encoding 20/09/2016 09:41:23 Client supports GII version 1 20/09/2016 09:41:23 Continuous updates enabled 20/09/2016 09:41:23 Continuous updates enabled Xlib: extension "GLX" missing on display ":2.0". Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. 20/09/2016 09:42:44 Client X.Y.Z.K gone 20/09/2016 09:42:44 Statistics: 20/09/2016 09:42:44 key events received 88, pointer events 1074 20/09/2016 09:42:44 framebuffer updates 1162, rectangles 25268, bytes 247197418 20/09/2016 09:42:44 LastRect markers 1113, bytes 13356 20/09/2016 09:42:44 cursor shape updates 42, bytes 86040 20/09/2016 09:42:44 CopyRect rectangles 11, bytes 176 20/09/2016 09:42:44 Tight rectangles 24102, bytes 247097846 20/09/2016 09:42:44 raw equivalent 3440.598900 Mbytes, compression ratio 13.924034

However, no errors are present on the server log when the problem happens.

Thank you

Thank you

[hildred]

I have not tried to do what you are doing, but I am willing to bet that the problem is that the game wants a mouse, not a tablet as it's pointing device.

Ok, let's start with some theory: a mouse reports relative motion, a little to the left, a little to the right. A tablet reports absolute position. The VNC protocol reports absolute position of the mouse pointer. In most cases the absolute cursor position is more important than relative so this works fine. In some games (particularly first person shooters) the relative motion is more important, however because you are using VNC it is not available and problems are to be expected.

The solution is to transmit relative position information for use by the game, but I cannot think of an easy way to do so. The simplest two ideas I have are to extend the protocol (there is some work in this direction) or to transmit relative motion in a parallel control channel (USB over ethernet may be a candidate) but both would require server code changes.

On Tue, Sep 20, 2016 at 1:46 AM, ntt1985 notifications@github.com wrote:

I'm using turbovnc + virtualgl with steam games and I have a problem with mouse during the game (I tried only with half life2 and transmission).

Used version: VNCSERVER: [ntt@fedora22 bin]$ ./vncserver --help TurboVNC Server v2.0.95 (build 20160909)

CLIENT: ntt@juniper:/opt/TurboVNC/bin$ ./vncviewer-java --help TurboVNC Viewer v2.0.95 (build 20160909) [JVM: amd64]

Basically the mouse works well during the vnc session, in the steam menu and in the game menu but when I start to play it becomes "crazy" and the game is out of control. If I press "esc" the mouse works normally and I can exit the game using the half life menu.

logs of vncviewer with -loglevel 150 during the game: http://pastebin.com/dYFpU5tN out of xinput list --long : http://pastebin.com/Y6zSFKL5 log of the vncserver:

20/09/2016 09:41:19 Got connection from client X.Y.Z.K 20/09/2016 09:41:19 Using protocol version 3.8 20/09/2016 09:41:19 Enabling TightVNC protocol extensions 20/09/2016 09:41:19 Advertising Tight auth cap 'VENCRYPT' 20/09/2016 09:41:19 Advertising Tight auth cap 'VNCAUTH_' 20/09/2016 09:41:19 Advertising Tight auth cap 'ULGNAUTH' 20/09/2016 09:41:20 Client requested VeNCrypt sub-type 258 20/09/2016 09:41:20 Deferring TLS handshake 20/09/2016 09:41:23 Full-control authentication enabled for X.Y.Z.K 20/09/2016 09:41:23 Pixel format for client X.Y.Z.K: 20/09/2016 09:41:23 32 bpp, depth 24, little endian 20/09/2016 09:41:23 true colour: max r 255 g 255 b 255, shift r 16 g 8 b 0 20/09/2016 09:41:23 no translation needed 20/09/2016 09:41:23 Enabling full-color cursor updates for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Desktop Size protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Extended Desktop Size protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 rfbProcessClientNormalMessage: ignoring unknown encoding -307 (fffffecd) 20/09/2016 09:41:23 Enabling LastRect protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Continuous Updates protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling Fence protocol extension for client X.Y.Z.K 20/09/2016 09:41:23 Enabling GII extension for client X.Y.Z.K 20/09/2016 09:41:23 Using tight encoding for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG subsampling 0, Q100 for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG quality 95 for client X.Y.Z.K 20/09/2016 09:41:23 Using JPEG subsampling 0 for client X.Y.Z.K 20/09/2016 09:41:23 Using Tight compression level 1 for client X.Y.Z.K 20/09/2016 09:41:23 Using 1 thread for Tight encoding 20/09/2016 09:41:23 Client supports GII version 1 20/09/2016 09:41:23 Continuous updates enabled 20/09/2016 09:41:23 Continuous updates enabled Xlib: extension "GLX" missing on display ":2.0". Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x4600008 (HALF-LIFE ) Window manager warning: meta_window_activate called by a pager with a 0 timestamp; the pager needs to be fixed. 20/09/2016 09:42:44 Client X.Y.Z.K gone 20/09/2016 09:42:44 Statistics: 20/09/2016 09:42:44 key events received 88, pointer events 1074 20/09/2016 09:42:44 framebuffer updates 1162, rectangles 25268, bytes 247197418 20/09/2016 09:42:44 LastRect markers 1113, bytes 13356 20/09/2016 09:42:44 cursor shape updates 42, bytes 86040 20/09/2016 09:42:44 CopyRect rectangles 11, bytes 176 20/09/2016 09:42:44 Tight rectangles 24102, bytes 247097846 20/09/2016 09:42:44 raw equivalent 3440.598900 Mbytes, compression ratio 13.924034

However, no errors are present on the server log when the problem happens.

Thank you

Thank you

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/TurboVNC/turbovnc/issues/67, or mute the thread https://github.com/notifications/unsubscribe-auth/AFul9kp8U0AxfVX0JbhewJX0tcUdDYplks5qr49ygaJpZM4KBUcC .

Ben Hildred Automation Support Services 303 815 6721

[dcommander]

I thought at first that this might be another regression caused by the remote X Input feature. The good news is that it isn't. Your logs confirm that the TurboVNC Viewer isn't trying to use that feature. Also, the issue is reproducible with older versions of TurboVNC, as well as with other VNC flavors. The mouse motion is interpreted properly by the game as long as the mouse remains within the confines of the VNC viewer window, but as soon as it reaches the edge, the VNC viewer will stop reporting movement in one direction (for instance, if you mouse beyond the right edge, then only vertical movement is reported until the mouse re-enters the window.) Hopefully it isn't necessary to add relative mouse motion support to RFB, although if that is necessary, then the GII extension that was implemented in order to support remote X input will allow us to straightforwardly do that (NOTE: "straightforward" doesn't necessarily mean "easy" or "fast". That would still be a significant project.) I am hoping that this may be fixable by simply adding a mode to the viewer that causes it to continue reporting mouse movement beyond the viewer window edges, although further study is warranted. Since this represents a new feature, unfortunately it isn't something I can fix for TurboVNC 2.1.

[dcommander]

Unfortunately we're not as lucky as I would have hoped. Normal mouse movement is simply not reported to the application once the mouse leaves the viewer window, and that seems to be the case not just for Java but for pretty much every windowing toolkit out there. So that presents us with a handful of problems:

• How to "capture" the mouse pointer and keep it confined to within the viewer window. There seem to be some ways of doing this, but they aren't very pretty. They generally involve activating some sort of "game-friendly" mouse mode, which forces the mouse pointer to the center of the window after every motion event. Obviously this would have to be switched on in the viewer when playing a game and switched off when doing anything else, which is far from ideal. The alternative would be to stipulate that games requiring relative mouse motion will never work properly unless the TurboVNC Viewer is run in full-screen mode.
• Extending the existing RFB GII implementation in TurboVNC to support relative valuators. This is at least straightforward, but it will require a lot of testing.
• Developing a platform-independent way of computing the relative valuators on the client end. On Linux, we can probably take advantage of the existing TurboVNC Helper library, which already has low-level access to X Input. How to do the same thing on Mac and Windows clients, however, is an open question. In particular, even when using full-screen mode, it's unclear how one would compute the relative valuators once the pointer is at the edge of the screen. At that point, the O/S will still send us motion events, but the coordinates will only change in the direction perpendicular to the window/screen edge. Hence, it may be necessary to develop some sort of "game-friendly" mouse mode, regardless of the full-screen stipulation.

As @hildred points out, this speaks to a fundamental difference between the needs of typical remote display applications and the needs of games. AFAIK, no remote display software currently does this, so we're kind of on the bleeding edge here.

[nickblock]

HI I am experiencing the same problem described above. We have an unreal project we are streaming over turbovnc and virtualgl.

I find that by down scaling the input by 100 times (in unreal) I get a clearer view of the mouse input. It looks like there is a long delay from the mouse input. By pulling left for sometime the app eventually responds to the left. Then you need to pull to the right for awhile, the app continues going left before eventually starting to pull right. It's like there is a large buffer holding the input values somewhere..

I noticed you have parked this issue for now. Could you give me a clue as to where In the turbovnc code I might explore this problem?

This behaviour does not happen running the app on windows via vnc. But other vnc servers exhibit this behaviour running from linux.

[nickblock]

Actually you're right. It's treating the screenpos as the mouse movement.

[dcommander]

@nickblock I'm open to anything that might improve the situation and could definitely use some assistance on the issue. Here are some relevant sections of the code:

Windows viewer: processing mouse input from the windowing system and translating it into rfbPointerEvent messages: https://github.com/TurboVNC/turbovnc/blob/master/win/vncviewer/ClientConnection.cpp#L2521-L2563 https://github.com/TurboVNC/turbovnc/blob/master/win/vncviewer/ClientConnection.cpp#L2741-L2887

Java/Mac/Un*x viewer: processing mouse input from the windowing system and translating it into rfbPointerEvent messages: https://github.com/TurboVNC/turbovnc/blob/master/java/com/turbovnc/vncviewer/DesktopWindow.java#L450-L480 https://github.com/TurboVNC/turbovnc/blob/master/java/com/turbovnc/vncviewer/CConn.java#L2142-L2200

Server: receiving rfbPointerEvent messages and translating them into X11 input https://github.com/TurboVNC/turbovnc/blob/master/unix/Xvnc/programs/Xserver/hw/vnc/rfbserver.c#L1248-L1267 https://github.com/TurboVNC/turbovnc/blob/master/unix/Xvnc/programs/Xserver/hw/vnc/kbdptr.c#L448-L486

[nickblock]

hey thanks for that dcommnader. I didn't find a solution in the turbovnc side of things. We got around it by preventing Unreal taking exclusive control of the mouse.

[dcommander]

If it's relying on relative mouse movements, then there isn't going to be an easy solution in TurboVNC. Referring to previous comments, it will require extensive modification to both the viewer and the server. The protocol-level modifications are at least well-understood and would involve leveraging the existing GII RFB extension, but the viewer modifications would require implementing some sort of game-friendly mouse mode whereby the mouse is confined to the viewer window. Virtualization solutions like Parallels Desktop implement such a "mouse capture" feature, so this would be similar in concept. Doable, but would require financial sponsorship or a well-written code contribution.

[jacmet]

Why not just support the existing QEMU pointer motion change extension instead of moving to GII? https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-pointer-motion-change-pseudo-encoding

[dcommander]

@jacmet I was just spitballing regarding GII, because I'm familiar with that extension and because it's already implemented in TurboVNC (we use it in order to provide support for Wacom tablets.) I wasn't aware of the QEMU pointer motion change extension until now, but I agree that that would be an easier solution to this problem than using GII.

[polhaghverdian]

Is there any other VNC that support relative mouse movements?

[dcommander]

To the best of my knowledge, no, there isn't.

[laseryuan]

RealVNC has "Enable direct capture mode" in server and "Relative Pointer Motion" for viewer: Running directly rendered apps such as Minecraft remotely

[dcommander]

OK, so amend my comment to say “there is no free VNC solution with that feature” (AFAIK).

[LeRoms77]

I hope there will be a solution for mouse problem. Meanwhile steam remote play is only solution i know

[dcommander]

It can be fixed, but it would take time to get the solution right, and since I'm an independent developer, time is money. Unfortunately, this class of problem (problems that primarily affect games) doesn't usually receive funding, since it doesn't affect corporate users, and the TurboVNC General Fund is usually exhausted on more mundane tasks like putting out releases and such.

[dcommander]

I did some further investigation into this. It seems straightforward to implement the QEMU Pointer Motion Change pseudo-encoding, although it would require a toggle in the TurboVNC Viewer much like the one that RealVNC uses. The primary reason for that is that the client operating system-- and, by extension, Java-- feeds us absolute mouse coordinates, so it will be necessary to compute the relative movement from those absolute coordinates. In order to do that, we have to ensure that the mouse never reaches the edge of the client desktop. Games tend to solve that problem by artificially confining the pointer to the center of the window. Probably enabling relative mouse movement should also hide the mouse cursor. One open question in my mind is how this would work with collaboration. If one connected viewer enables relative pointer motion, should it be automatically enabled for all other viewers?

[faust93]

I hope there will be a solution for mouse problem. Meanwhile steam remote play is only solution i know

It's quite easy to implement on the server side even without fully implementing QEMU Pointer encoding. I have working implementation in my TurboVNC fork. I'm using basically the same approach like QEMU Pointer does, masking X&Y with 0x7FFF in relative mode for the server to distinguish between absolute and relative: https://github.com/faust93/turbovnc/commit/a92ce21ae80ae9c86633837857cac94406f901e2

The most job to do is on the client side. First, there should be some switch to toggle absolute/relative mode, mouse X&Y coordinates are just masked with 0x7FFF in relative mode so the server understands that provided coordinates are relative. Next, the client have to handle mouse correctly in relative mode and restrict it by the window borders. I have my own VNC client implementation based on SDL and it was quite easy to implement relative mouse since SDL provides that functionality natively (SDL_SetRelativeMouseMode) so I just have to notice mouse mode toggle and provide masked relative coordinates to the server: https://github.com/faust93/bwVNC/blob/3215aa53fb6736183a3345c5c633691fb8106e0a/bwvnc/bwvnc.c#L455 Now able to play FPS games remotely ;)

[dcommander]

@faust93 The problem is that your approach violates the RFB protocol spec, which is why it will be necessary to use a defined RFB extension. It's not a huge deal to use an RFB extension. As you pointed out, the complexity will be mostly in the viewer, which is why this is marked "funding needed." I may need to obtain the relative mouse coordinates through JNI rather than relying on Java.

[TechArtistG]

+1 and thanks for all the discussion, just ran into this today using Unreal Editor

[TechArtistG]

Haven't tried it but this could be an interim fix https://yingtongli.me/blog/2019/11/18/input-over-ssh.html

[einhander]

OK, so amend my comment to say “there is no free VNC solution with that feature” (AFAIK).

There is KasmVNC https://github.com/kasmtech/KasmVNC, it has an option to switch input methods. KasmVNC server not compatible with any vncviewer except its own.

[dcommander]

Yes, since I made my comment about there being no free solution, TigerVNC implemented this feature, and the KasmVNC Server is based on the TigerVNC Server. The KasmVNC Viewer is based on noVNC. To the best of my understanding, they are both open source and open spec, but they use RFB security extensions that aren't implemented by other VNC flavors. It should theoretically be possible to make another VNC viewer compatible with Kasm, but it would be a hell of a lot easier to add relative mouse movement to TurboVNC. I haven't tested TigerVNC's implementation, but it would be useful if others would test it and give me feedback as to whether it is sufficient to solve the problem. If so, then straightforwardly porting their implementation would be easier than coming up with our own implementation from scratch.