search

ponty / framebuffer-vncserver

VNC server for Linux framebuffer devices
GNU General Public License v2.0
169 stars 69 forks source link

Multitouch support ? #6

Closed maleselo closed 4 years ago

maleselo commented 4 years ago

Hi,

I am trying to run a simple Qt application (a simple button that changes a rectangle color) on my system (Buildroot, iMX6 and ILI251x touchscreen). I would like to remotely access this application from a VNC client (visualize and interact with).

I tried framebuffer-vncserver and vncviewer and I successfully visualize my application on the client. But I could not remotely push my button.

Input events from vncwiever generate ABS_X, ABS_Y and BTN_TOUCH events on the server side but they are not recognized and my button is not pushed as it should be.

root@target:# evtest /dev/input/event1 &
root@target:# ./framebuffer-vncserver -f /dev/fb1 -t /dev/input/ev
ent1 -v &
root@target:# Initializing framebuffer device /dev/fb1...
  xres=800, yres=600, xresv=800, yresv=600, xoffs=0, yoffs=0, bpp=32
  offset:length red=16:8 green=8:8 blue=0:8
No keyboard device
Initializing touch device /dev/input/event1 ...
  x:(0 14800)  y:(0 9800)
Initializing VNC server:
        width:  800
        height: 600
        bpp:    32
        port:   5900
        rotate: 0
Initializing server...
01/01/2000 03:11:27 Listening for VNC connections on TCP port 5900
01/01/2000 03:11:27 Listening for VNC connections on TCP6 port 5900
01/01/2000 03:11:32 Got connection from client 192.168.214.248
01/01/2000 03:11:32   other clients:
01/01/2000 03:11:32 Normal socket connection
01/01/2000 03:11:32 Client Protocol Version 3.8
01/01/2000 03:11:32 Protocol version sent 3.8, using 3.8
  fps: 0.000000
Dirty page: 801x600+0+0...
01/01/2000 03:11:32 rfbProcessClientSecurityType: executing handler for type 1
01/01/2000 03:11:32 rfbProcessClientSecurityType: returning securityResult for client rfb version >= 3.8
Got ptrevent: 0000 (x=400, y=300)
01/01/2000 03:11:32 Pixel format for client 192.168.214.248:
01/01/2000 03:11:32   8 bpp, depth 6
01/01/2000 03:11:32   true colour: max r 3 g 3 b 3, shift r 4 g 2 b 0
01/01/2000 03:11:32 Enabling full-color cursor updates for client 192.168.214.248
01/01/2000 03:11:32 Enabling NewFBSize protocol extension for client 192.168.214.248
01/01/2000 03:11:32 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFECC)
01/01/2000 03:11:32 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFECD)
01/01/2000 03:11:32 Enabling LastRect protocol extension for client 192.168.214.248
01/01/2000 03:11:32 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC7)
01/01/2000 03:11:32 rfbProcessClientNormalMessage: ignoring unsupported encoding type Enc(0xFFFFFEC8)
01/01/2000 03:11:32 Using compression level 1 for client 192.168.214.248
01/01/2000 03:11:32 Using image quality level 0 for client 192.168.214.248
01/01/2000 03:11:32 Using JPEG subsampling 1, Q15 for client 192.168.214.248
01/01/2000 03:11:32 Using raw encoding for client 192.168.214.248
...
Got ptrevent: 0001 (x=128, y=82)
Event: time 946696300.137120, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 1
Event: time 946696300.137120, type 3 (EV_ABS), code 0 (ABS_X), value 2368
Event: time 946696300.137120, type 3 (EV_ABS), code 1 (ABS_Y), value 1339
Event: time 946696300.137120, -------------- SYN_REPORT ------------
injectTouchEvent (screen(128,82) -> touch(2368,1339), down=1)
Got ptrevent: 0000 (x=128, y=82)
injectTouchEvent (screen(128,82) -> touch(2368,1339), down=0)
Event: time 946696300.170415, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 0

When I press the button directly with the touchscreen (not with the VNC client), I have got these logs:

root@target:# Event: time 946696796.496230, type 3 (EV_ABS), code 47 (ABS_MT_SLOT), value 0
Event: time 946696796.496230, type 3 (EV_ABS), code 57 (ABS_MT_TRACKING_ID), value 102
Event: time 946696796.496230, type 3 (EV_ABS), code 53 (ABS_MT_POSITION_X), value 1109
Event: time 946696796.496230, type 3 (EV_ABS), code 54 (ABS_MT_POSITION_Y), value 660
Event: time 946696796.496230, type 3 (EV_ABS), code 47 (ABS_MT_SLOT), value 5
Event: time 946696796.496230, type 3 (EV_ABS), code 57 (ABS_MT_TRACKING_ID), value -1
Event: time 946696796.496230, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 1
Event: time 946696796.496230, type 3 (EV_ABS), code 0 (ABS_X), value 1109
Event: time 946696796.496230, type 3 (EV_ABS), code 1 (ABS_Y), value 660
Event: time 946696796.496230, -------------- SYN_REPORT ------------
Event: time 946696796.585903, type 3 (EV_ABS), code 47 (ABS_MT_SLOT), value 0
Event: time 946696796.585903, type 3 (EV_ABS), code 57 (ABS_MT_TRACKING_ID), value -1
Event: time 946696796.585903, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 0
Event: time 946696796.585903, -------------- SYN_REPORT ------------

It looks like framebuffer-vncserver does not generate all the required events to actually push my button. I did some modifications on the touch.c file in order to generate these events (ABS_MT_TRACKING_ID, ABS_MT_POSITION_X and ABS_MT_POSITION_Y):

diff --git a/framebuffer-vncserver.pro b/framebuffer-vncserver.pro
index fe49b51..48f778b 100644
--- a/src/touch.c
+++ b/src/touch.c
@@ -28,6 +28,7 @@ static int touchfd = -1;
 static int xmin, xmax;
 static int ymin, ymax;
 static int rotate;
+static int trkg_id = -1;

 int init_touch(const char *touch_device, int vnc_rotate)
 {
@@ -101,6 +102,36 @@ void injectTouchEvent(int down, int x, int y, struct fb_var_screeninfo *scrinfo)

     if (down >= 0)
     {
+        // Then send a ABS_MT_TRACKING_ID
+        gettimeofday(&ev.time, 0);
+        ev.type = EV_ABS;
+        ev.code = ABS_MT_TRACKING_ID;
+        ev.value = ++trkg_id;
+        if (write(touchfd, &ev, sizeof(ev)) < 0)
+        {
+            error_print("write event failed, %s\n", strerror(errno));
+        }
+
+        // Then send a ABS_MT_POSITION_X
+        gettimeofday(&ev.time, 0);
+        ev.type = EV_ABS;
+        ev.code = ABS_MT_POSITION_X;
+        ev.value = xin;
+        if (write(touchfd, &ev, sizeof(ev)) < 0)
+        {
+            error_print("write event failed, %s\n", strerror(errno));
+        }
+
+        // Then send a ABS_MT_POSITION_Y
+        gettimeofday(&ev.time, 0);
+        ev.type = EV_ABS;
+        ev.code = ABS_MT_POSITION_Y;
+        ev.value = yin;
+        if (write(touchfd, &ev, sizeof(ev)) < 0)
+        {
+            error_print("write event failed, %s\n", strerror(errno));
+        }
+
         // Then send a BTN_TOUCH
         gettimeofday(&ev.time, 0);
         ev.type = EV_KEY;
@@ -110,6 +141,16 @@ void injectTouchEvent(int down, int x, int y, struct fb_var_screeninfo *scrinfo)
         {
             error_print("write event failed, %s\n", strerror(errno));
         }
+    } else {
+        // ABS_MT_TRACKING_ID
+        gettimeofday(&ev.time, 0);
+        ev.type = EV_ABS;
+        ev.code = ABS_MT_TRACKING_ID;
+        ev.value = -1;
+        if (write(touchfd, &ev, sizeof(ev)) < 0)
+        {
+            error_print("write event failed, %s\n", strerror(errno));
+        }
     }

     // Then send the X

With these modifications, when I push the button from the VNC client, the push is well recognized by my Qt application (I do have some multi touch x and y position issues though).

I am pretty unaware of how input devices works on Linux as well as VNC, so any suggestion will be appreciated. I would like to know if you plan to support multi touchscreen events like ABS_MT_TRACKING_ID, ABS_MT_POSITION_X and ABS_MT_POSITION_Y ?

Best regards, Maleselo

ponty commented 4 years ago

I added your patch, then fixed mouse grab and mouse release after reading the doc: https://www.kernel.org/doc/Documentation/input/multi-touch-protocol.txt

I tested it with single-touch and multi-touch devices on iMX6 (QT demo CinematicExperience, Mouse press, release, grab.) Can you test if it works on your hardware?

maleselo commented 4 years ago

Hi @ponty, Thanks for your very quick answer.

I still have some issues. The ABS_MT_POSITION_X and ABS_MT_POSITION_Y events generated by the framebuffer-vncserver have different values from ABS_X and ABS_Y:

Got ptrevent: 0000 (x=148, y=118)
Got ptrevent: 0001 (x=148, y=118)
Event: time 946687240.090812, type 3 (EV_ABS), code 57 (ABS_MT_TRACKING_ID), value 14
Event: time 946687240.090812, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 1
injectTouchEvent (screen(148,118) -> touch(2738,1927), mouse=1)
Event: time 946687240.090812, type 3 (EV_ABS), code 53 (ABS_MT_POSITION_X), value 148
Event: time 946687240.090812, type 3 (EV_ABS), code 54 (ABS_MT_POSITION_Y), value 118
Event: time 946687240.090812, type 3 (EV_ABS), code 0 (ABS_X), value 2738
Event: time 946687240.090812, type 3 (EV_ABS), code 1 (ABS_Y), value 1927
Event: time 946687240.090812, -------------- SYN_REPORT ------------
Got ptrevent: 0000 (x=148, y=118)
Event: time 946687240.213149, type 3 (EV_ABS), code 57 (ABS_MT_TRACKING_ID), value -1
injectTouchEvent (screen(148,118) -> touch(2738,1927), mouse=0)
Event: time 946687240.213149, type 1 (EV_KEY), code 330 (BTN_TOUCH), value 0
Event: time 946687240.213149, -------------- SYN_REPORT ------------
Got ptrevent: 0000 (x=148, y=118)

I tried this and it works well for me. What do you think about it ?

diff --git a/src/touch.c b/src/touch.c
index 14716f8..862af5d 100644
--- a/src/touch.c
+++ b/src/touch.c
@@ -156,7 +156,7 @@ void injectTouchEvent(enum MouseAction mouseAction, int x, int y, struct fb_var_
     gettimeofday(&ev.time, 0);
     ev.type = EV_ABS;
     ev.code = ABS_MT_POSITION_X;
-    ev.value = xin;
+    ev.value = x;
     if (write(touchfd, &ev, sizeof(ev)) < 0)
     {
         error_print("write event failed, %s\n", strerror(errno));
@@ -166,7 +166,7 @@ void injectTouchEvent(enum MouseAction mouseAction, int x, int y, struct fb_var_
     gettimeofday(&ev.time, 0);
     ev.type = EV_ABS;
     ev.code = ABS_MT_POSITION_Y;
-    ev.value = yin;
+    ev.value = y;
     if (write(touchfd, &ev, sizeof(ev)) < 0)
     {
         error_print("write event failed, %s\n", strerror(errno));

Here is my evtest output just in case:

evtest /dev/input/event1
Input driver version is 1.0.1
Input device ID: bus 0x18 vendor 0x0 product 0x0 version 0x0
Input device name: "ili251x Touchscreen"
Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 59 (KEY_F1)
    Event code 60 (KEY_F2)
    Event code 61 (KEY_F3)
    Event code 62 (KEY_F4)
    Event code 330 (BTN_TOUCH)
  Event type 3 (EV_ABS)
    Event code 0 (ABS_X)
      Value   3829
      Min        0
      Max    14800
    Event code 1 (ABS_Y)
      Value   2156
      Min        0
      Max     9800
    Event code 47 (ABS_MT_SLOT)
      Value      5
      Min        0
      Max        5
    Event code 53 (ABS_MT_POSITION_X)
      Value      0
      Min        0
      Max    14800
    Event code 54 (ABS_MT_POSITION_Y)
      Value      0
      Min        0
      Max     9800
    Event code 57 (ABS_MT_TRACKING_ID)
      Value      0
      Min        0
      Max    65535
Properties:
Testing ... (interrupt to exit)
ponty commented 4 years ago

This seems to be correct.

maleselo commented 4 years ago

Thank you very much