Use with Surface Pro Touch Screen

[Neightro]

I discovered this project while looking for a solution for touchscreen gestures on my Surface Pro 6. While it works fantastically on the trackpad, nothing changes about the behavior of the touch screen.

The touch screen currently functions inside of Plasma, and all of my fingers are recognized within libinput debug-gui. Here is the device as outputted by libinput list-devices:

Device:           IPTS Touch
Kernel:           /dev/input/event13
Group:            5
Seat:             seat0, default
Size:             259x171mm
Capabilities:     touch
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Accel profiles:   n/a
Rotation:         n/a

Other specs Version: git master; brand new install Kernel: 5.8.6-arch1-1-surface. Desktop environment: KDE Plasma Please let me know if there is any more information I can provide.

Is there anything that I can do right now to get gesture support to work with my touch screen? In the case that the device isn't supported right now, I hope that you will consider it. I was excited to find this project; it is the only one I've come across so far that explicitly mentions touch screens as one of its use cases, so I'm hopeful. :)

[natask]

@Neightro I suspect what is going on is that your touchscreen isn't being matched. parsing for touchscreens is currently done on string "touchscreen" and "400 0 0 0 0 0". I have been trying to find a general parsing mechanism to match touchscreens. I will add "IPTS Touch" to the list for now. The latest commit should work for you.

would you mind reporting cat /proc/bus/input/devices for your touchscreen? That will help me find a general parsing mechanism to match touchscreens.

I wonder how libinput does its matching.

[Neightro]

Thanks for getting back to me! I'll test the new commit as soon as possible, and I'll update here to report on it. I'm assuming that the commit you're referring to is on master?

Strangely, /proc/bus/input/devices appears to be empty for me. I made sure I was running linux-surface, ran Plasma to check that the touch screen was working, and ran ls as root. I wish that I understood more about this to say why that's the case, but I'm really not sure.

Edit: I just tried pulling master and running the install script; touchpad still works, but it doesn't seem to be recognizing my touch screen. As an aside, I recall Fusuma (or one of the gesture tools I tried before this one, at any rate) had the ability to list the devices it recognized. If this project had something similar, that might give a better indication as to what is happening.

[natask]

Yes I was referring to pulling master and re-installing the script. apologies for the vagueness.

Can you try running gestures debug. It now prints matched touchpad and touchscreen event numbers. try executing touchscreen gestures and see if it reports anything.

This project reads the file /proc/bus/input/devices to get touchpads and touchscreens. It is strange that the file /proc/bus/input/devices is empty. I am surprised that touchpad is even working.

I believe the script should be detecting your touchscreen with my previous modification because your touchscreen device name " IPTS Touch" has been added to the device name matchlist. Given that touchscreen gestures are still not working, my assumption is that touchscreen was being matched even before but the program terminated (runs on a different thread) because of a configuration issue.

From your message maybe you ran ls /proc/bus/input/devices instead of cat /proc/bus/input/devices? try cat /proc/bus/input/devices if you didn't already do so.

[Neightro]

You were right; I did mistakenly run ls instead of cat; I must have been thinking about the way /dev/ works. The output for IPTS Touch is right below. Sorry about that, and thanks for your patience. 😅

I: Bus=0006 Vendor=045e Product=001f Version=0000
N: Name="IPTS Touch"
P: Phys=
S: Sysfs=/devices/virtual/input/input30
U: Uniq=
H: Handlers=event20
B: PROP=2
B: EV=9
B: ABS=260800000000000

I just tried installing the latest version on master and running gestures.debug. Here is the output when I first run it:

❯ gestures debug
True
Handlers for touchpads: ['Handlers=event6']
Handlers for touchscreens:  ['Handlers=event20']
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/bin/gestures", line 685, in test
    w = Worker(q, all_gestures[dev])
KeyError: 'touchscreen|IPTS touch'

It looks like it recognizes the touchscreen properly at least. Performing actions with the touch pad causes messages to print to stdout, though that's not the case for the screen. As you'd probably expect the exception happens when running gestures execute as well.

Edit: If it's helpful at all to know, the exception does not occur while running the vanilla Arch Linux kernel, which causes the touch screen to not appear as a device. It must be related to the touch screen code in some way.

[natask]

@Neightro check it out now. It should work.

[Neightro]

The good news is that gestures debug now prints messages while touching the screen! The bad news is that it seems like there is still one exception of some kind; it occurs immediately after releasing my finger off the screen for the first time. Afterward nothing prints when I touch the screen.

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
{'type': 'pinch_o', 'total': {'x-cum': 188.0, 'y-cum': 101.5, 'moved': 2, 'dis-cum': -40}, 'slots': {'0': {'x-cum': -224.0, 'y-cum': -536.5, 'moved': True}, '1': {'x-cum': 500.0, 'y-cum': 541.5, 'moved': True}}}
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/bin/gestures", line 737, in test
    status_dict[slot]["x"] = float(state) / factor * orientation_x;
KeyError: '0'

[natask]

That is very strange. Does this always happen? I have added more debugging prints. Can you try the latest commit using only touchscreen gestures and getting the log when it breaks? That will help me get to the bottom of this.

Also can you show me what keycodes your touchscreen generates. maybe keycodes "(ABS_MT_TRACKING_ID) or (ABS_MT_SLOT) are something else on your touchscreen since it is a bug with slots. Selecting your device under evtest shows the keycodes.

for example for me it is

$ evtest
No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event7:      ipts 1B96:005E
Select the device event number [0-24]: 7
Input driver version is 1.0.1
Input device ID: bus 0x44 vendor 0x1b96 product 0x5e version 0x0
Input device name: "ipts 1B96:005E"
Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 330 (BTN_TOUCH)
  Event type 3 (EV_ABS)
    Event code 0 (ABS_X)
      Value   8028
      Min        0
      Max     9600
      Resolution      34
    Event code 1 (ABS_Y)
      Value   3105
      Min        0
      Max     7200
      Resolution      38
    Event code 47 (ABS_MT_SLOT)
      Value      2
      Min        0
      Max        9
    Event code 48 (ABS_MT_TOUCH_MAJOR)
      Value      0
      Min        0
      Max     9600
      Resolution      34
    Event code 49 (ABS_MT_TOUCH_MINOR)
      Value      0
      Min        0
      Max     7200
      Resolution      38
    Event code 52 (ABS_MT_ORIENTATION)
      Value      0
      Min        0
      Max        1
    Event code 53 (ABS_MT_POSITION_X)
      Value      0
      Min        0
      Max     9600
      Resolution      34
    Event code 54 (ABS_MT_POSITION_Y)
      Value      0
      Min        0
      Max     7200
      Resolution      38
    Event code 55 (ABS_MT_TOOL_TYPE)
      Value      0
      Min        0
      Max        2
    Event code 57 (ABS_MT_TRACKING_ID)
      Value      0
      Min        0
      Max    65535
    Event code 60 (ABS_MT_TOOL_X)
      Value      0
      Min        0
      Max     9600
      Resolution      34
    Event code 61 (ABS_MT_TOOL_Y)
      Value      0
      Min        0
      Max     7200
      Resolution      38
  Event type 4 (EV_MSC)
    Event code 5 (MSC_TIMESTAMP)
Properties:
  Property type 1 (INPUT_PROP_DIRECT)
Testing ... (interrupt to exit)

[natask]

I advice updating your touchscreen configurations if you use the xdotool version. 3 finger tap on touchscreen is now correctly mapped to paste (shift + Insert).

[Neightro]

Sure thing! Here is the debug output from a fresh installation of gestures, while performing a two-finger pinch (the test I've been using throughout this issue). The output is rather long; I could have trimmed much of it, but thought it best not to.

gestures-output.txt

Here is my output from evtest while performing another two-finger pinch.

evtest-output.txt

I appreciate the note about the config. I believe that I'm using the evemu version, though I'm not completely sure. I don't have any custom configs yet; it's been helpful just to use the default while I test. Does the latest backup still get copied if I say no to the backup, by the way?

[natask]

Thanks for the logs. I will investigate them.

The install script only makes a backup if you tell it to install a config when there is already another config installed.

[natask]

I understand what is causing the issue. Your touchscreen unlike mine releases stray (ABS_MT_POSITION_X) == 0 and (ABS_MT_POSITION_Y) == 0 after signaling that a finger is removed.

This should be fixed in the latest commit.

[Neightro]

It's working; thank you so much for your help! I was able to test by swiping three fingers upward, which for me just writes an 'i' in my terminal. The key combo is probably just not mapped to anything in Plasma.

There still seem to be some minor problems, such as the two-finger pinch not working with the touch screen. These things are perhaps best off as their own issues, though.

I did come across this exception, which I will include here for now. I haven't worked out when it tends to happen, and gestures debug scrolls too quickly for me to be able to notice it. I can either create a new issue or post on this thread when I figure out more details; whichever you prefer.

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/bin/gestures", line 754, in test
    status_dict[slot]["x"] = float(state) / factor * orientation_x;
KeyError: '1'

[natask]

It was a bug introduced with the latest change. Can you try the latest commit. Thanks for sticking along.

[Neightro]

No problem; it's been fun, and I get to use this awesome program! I'm still managing to trigger the exception on the latest commit, and I know how to trigger it reliably now: it seems to happen when performing a three-finger touch.

I was able to capture some logs! I had gestures execute &>... in one terminal, and evtest running in another; I switched between them and started and stopped commands with alt+tab. I then touched the screen with three fingers, three times.

gestures-thread-3.txt

evtest-error.txt

Strangely, the stack trace printed in the terminal that was running evtest. I just copied and pasted that output rather than redirecting, so it will be in the evtest output.

[natask]

Does the script register touchscreen commands and execute them after the exception? I wasn't able to find where the exception happens within the log. Would you post another log, this time only with the output of the script? It now prints commands it reads from evtest during debug mode so the log for evtest isn't needed.

[Neightro]

I had been running gestures debug and evtest in two separate terminals, and for some reason the exception happened in evtest's shell, so that's where the exception ended up, unfortunately. I've been trying to trigger it again, and so far haven't succeeded. I'll try again later, and I'll update if I have a change in fortune.

In the meantime, the two-finger pinch doesn't seem to work on my touch screen. Is there any debug info or any logs I could get you to help with that?

Not a complaint to you, but amusingly I somehow got evtest to spam my shell history with its output, even on plain old bash. I should really go upstream that bug, but seeing as I can't reproduce that one either, I'm not sure how helpful that would be to them. Just my luck today, I guess. :)

[natask]

Great to hear it works for you now.

I am not sure why the exception output showed up on evtest's shell. my speculation is that the exception text was in your selection clipboard and with a 3 finger tap gesture simulated middle click selection paste into the evtest shell.

I haven't configured two-finger pinch in and out for touchscreen in the default configurations because most applications I use support that out the box.

If you would like to enable two-finger pinch in and out for touchscreen, you would need to change the commands executed. to do that, do the following.

• open up ~/.config/gestures.conf
• go to the section on touchscreen configuration (searching for touchscreen should get you there).
• now go to two-finger pinch section within touchscreen configuration replace

  # 2 finger pinch 
  '2': {
  'i' : {'start': ['', ''], 'update': {'i': [''], 'o': ['']}, 'end': [''], 'rep': ''},
  'o' : {'start': ['', ''], 'update': {'i': [''], 'o': ['']}, 'end': [''], 'rep': ''}
  },

  with

  # 2 finger pinch 
  '2': {
  'i' : {'start': ['evemu_do keydown control', 'evemu_do equal'], 'update': {'i': ['evemu_do plus'], 'o': ['evemu_do minus']}, 'end': ['evemu_do keyup ctrl'], 'rep': ''},
  'o' : {'start': ['evemu_do keydown control', 'evemu_do minus'], 'update': {'i': ['evemu_do plus'], 'o': ['evemu_do minus']}, 'end': ['evemu_do keyup ctrl'], 'rep': ''}
  },

  more info in the configuration section.

feel free to add more logs if you are met with more errors.

currently the touchscreen pointer isn't disabled during gestures. This has the effect of moving the pointer while executing a gesture. I am thinking of disabling the pointer for the duration of gestures. So be on the lookout for an update.