Surface Pencil not usable in portrait mode due interchanged x and y coordinates

[ArPhil]

In noticed that since one of the last linux surface / iptsd updates (do not know the exact one), the surface pencil cannot be properly used in protrait mode anymore, since the x and y coordinate are interchanged:

• When moving the pen to the left, the cursor goes up (or down, depending on the orientation in portrait mode)
• When moving the pen to top, the cursor goes left (or right, depending on the orientation in portrait mode)

It seems, this cannot be corrected via an xinput coordinate transformation because wayland is used.

How can one fix this with the current linux-srface kernel / iptsd package?

(touch inputs in portrait mode work fine)

[StollD]

Rotating the stylus inputs is done by the desktop environment. iptsd doesnt take the rotation into account at all, just like all other linux input drivers.

What distribution and desktop environment are you using? Could you try to find out if this was somehow triggered by an iptsd update? You should be able to go back to iptsd 1.0.0, anything older won't be compatible with the current kernel drivers.

It does sound weird to me that the touchscreen rotates but the stylus doesnt. What happens when you flip the display by 180°?

[ArPhil]

Good morning and thanks for the fast reply! Sure, the details of my device:

• Surface Pro 5 2017 (with LTE)
• Ubuntu 22.04.2 LTS
• GNOME 42.9 with Wayland
• 6.4.7-surface-3 (it confuses me, that the kernel name says surface-3 although I have surface pro 5, but maybe this means nothing)

What happens when you flip the display by 180°? --> when using the pen, the cursor reacts opposite to expectation (when moving up, cursor goes down. when moving left, cursor goes right. and vice versa)

Could you try to find out if this was somehow triggered by an iptsd update? --> I was only able to get back to iptsd 1.2.1 by using the build (debian-latest) from actions here. Older versions like 1.1.1 were expired. When installing 1.2.1, the effect is the same as described earlier (axis interchanged in portrait mode).

Maybe my conclusion is wrong, but I believe that the kernel driver responsible for detecting and reacting to screen orientation changes is this one: /lib/modules/6.4.7-surface/kernel/drivers/hid/ipts

When I remove this driver folder (temporarily), and reboot, no touch input nor screen orientation changes occur. So I am not sure if it is correct, that no driver is responsible for the rotation / touch events because to me it seems like this one is. But maybe I am misinterpreting here something. When I put back the folder to the path mentioned above, touch inputs and screen rotation work again, though the issue with the pencil still exists.

I hope this helps in order to get an idea what could be wrong? In case I should try out something else, please let me know and I will happily do so.

[StollD]

• 6.4.7-surface-3 (it confuses me, that the kernel name says surface-3 although I have surface pro 5, but maybe this means nothing)

Its the third build of 6.4.7.

Older versions like 1.1.1 were expired. When installing 1.2.1, the effect is the same as described earlier (axis interchanged in portrait mode).

There are packages on GitHub releases that won't expire: https://github.com/linux-surface/iptsd/releases

Maybe my conclusion is wrong, but I believe that the kernel driver responsible for detecting and reacting to screen orientation changes is this one: /lib/modules/6.4.7-surface/kernel/drivers/hid/ipts

Yes, that the is the kernel driver for the touchscreen. But multiple things to consider:

• This driver only forwards raw touchscreen data to userspace, where iptsd processes it. The input devices for touchscreen and stylus are created by iptsd, not by the kernel driver.
• No linux input driver reacts to orientation changes. Every driver will always output the same coordinates. It is the job of the desktop environment to detect orientation changes and remap the coordinates accordingly.

In case I should try out something else, please let me know and I will happily do so.

Do you have libwacom-surface installed? If yes, check the Wacom entry in the GNOME Settings app to see if some option might be set wrong.

Were there any updates to the mutter package recently? That is the GNOME compositor, the component that is responsible for input handling and orientation changes.

[ArPhil]

Thanks again (also for the hints.

There are packages on GitHub releases that won't expire: https://github.com/linux-surface/iptsd/releases

Thanks. I tried iptsd 1.1.1 --> same situation / no change

Do you have libwacom-surface installed?

It seems like I do, but can´t remember to have installed these manually:

libwacom-bin-surface/unknown,now 2.7.0-3 amd64 [installed,automatic] libwacom-common-surface/unknown,now 2.7.0-3 amd64 [installed,automatic] libwacom-surface/unknown,now 2.7.0-3 amd64 [installed] libwacom2-surface/unknown,now 2.7.0-3 amd64 [installed,automatic] libwacom9-surface/unknown,now 2.7.0-3 amd64 [installed,automatic] xserver-xorg-input-wacom/jammy,now 1:1.0.0-3ubuntu1 amd64 [installed,automatic]

If yes, check the Wacom entry in the GNOME Settings app to see if some option might be set wrong.

I checked the "Wacom Tablet" option in Gnome Settings. The "Stylus" Tab says "No stylus found. Please move your stylus to the proximity of the tablet to configure it."

However, using the pencil still works in landscape mode as it should. (Never configured anything in the wacom options.)

Were there any updates to the mutter package recently?

It seems, that I did not have mutter installed at all: sudo apt policy mutter

mutter: Installed: (none) Candidate: 42.9-0ubuntu1 Version table: 42.9-0ubuntu1 500 500 http://de.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages 42.0-3ubuntu2 500 500 http://de.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

I checked for other packages: apt list --installed | grep mutter

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

gir1.2-mutter-10/jammy-updates,now 42.9-0ubuntu1 amd64 [installed,automatic] libmutter-10-0/jammy-updates,now 42.9-0ubuntu1 amd64 [installed,automatic] mutter-common/jammy-updates,jammy-updates,now 42.9-0ubuntu1 all [installed,automatic]

I installed the mutter package manually and restarted / rebooted, but this did also not help.

Would it be safe to uninstall any wacom related packages, since according to what you said, this is rather optional?

[StollD]

What does running libwacom-list-local-devices in a terminal output? Also, could you upload your dmesg?

[ArPhil]

Sure!

libwacom-list-local-devices: devices:

• name: 'Microsoft Surface Pro 6' bus: 'unknown' vid: '0x045e' pid: '0x001f' nodes:

  • /dev/input/event30: 'IPTS Stylus'

  Please note, that I have a surface pro 5, and not 6...

Please find the dmesg output as file attachment. (The last line is after I disconnected the type cover and rotated the tablet in portrait mode in order to see if I got further messages): dmesgrot.txt

At least in the dmesg output I can also see, that the stylus was detected, which surprises me, since the wacom settings claimed that there is none.

[StollD]

Please note, that I have a surface pro 5, and not 6...

MS did reuse components like the digitizer or even the entire mainboard on different models around that time. So its not something to worry about.

However, I really have no idea what is going on. libwacom detects a device, but then the gnome settings app should also at least show an entry for it since it is based on libwacom.

I also dont really understand why it is only listing the stylus device and not the touchscreen as well. But I guess that is normal, because the same happens for me on Fedora KDE and everything works fine.

One last idea, have you tried if it works on Xorg?

[ArPhil]

MS did reuse components like the digitizer or even the entire mainboard on different models around that time. So its not something to worry about.

Interesting. That is nice to know. :)

One last idea, have you tried if it works on Xorg?

I changed to Xorg during login and confirmed the X11 session via echo $XDG_SESSION_TYPE --> no changes. Also before I started noticing the issues with the pencil in portrait mode, it used to work fine with wayland. So being wayland as a reason for the issue is something I believe can be excluded.

What is interesting to me, is that i tried to remove all iptsd related elements via sudo apts purge iptsd and sudo dpkg --purge iptsd and even searched manually for anything iptsd related via sudo find / -name iptsd.

But even after several reboots and shutdowns after executing the commands above, the touchinput for fingers still work, meaning I can use my fingers with the touchscreen (but not the surface pen). Doesn`t this contradict to what you said earlier?:

This driver only forwards raw touchscreen data to userspace, where iptsd processes it. The input devices for touchscreen and stylus are created by iptsd, not by the kernel driver.

I tried to investigate and installed libinput-tools and executed libinput list-devices. One of the devices listed was:

Device:           IPTS 045E:001F Touchscreen
Kernel:           /dev/input/event27
Group:            4
Seat:             seat0, default
Size:             293x165mm
Capabilities:     touch 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Accel profiles:   n/a
Rotation:         n/a

I tried to get additional info via udevadm info -a -p $(udevadm info -q path -n /dev/input/event27) and got

Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:16.4/0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04/0000:045E:001F.0003/input/input36/event27':
    KERNEL=="event27"
    SUBSYSTEM=="input"
    DRIVER==""
    ATTR{power/async}=="disabled"
    ATTR{power/control}=="auto"
    ATTR{power/runtime_active_kids}=="0"
    ATTR{power/runtime_active_time}=="0"
    ATTR{power/runtime_enabled}=="disabled"
    ATTR{power/runtime_status}=="unsupported"
    ATTR{power/runtime_suspended_time}=="0"
    ATTR{power/runtime_usage}=="0"

  looking at parent device '/devices/pci0000:00/0000:00:16.4/0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04/0000:045E:001F.0003/input/input36':
    KERNELS=="input36"
    SUBSYSTEMS=="input"
    DRIVERS==""
    ATTRS{capabilities/abs}=="3"
    ATTRS{capabilities/ev}=="1b"
    ATTRS{capabilities/ff}=="0"
    ATTRS{capabilities/key}=="400 0 0 0 0 0"
    ATTRS{capabilities/led}=="0"
    ATTRS{capabilities/msc}=="10"
    ATTRS{capabilities/rel}=="0"
    ATTRS{capabilities/snd}=="0"
    ATTRS{capabilities/sw}=="0"
    ATTRS{id/bustype}=="0000"
    ATTRS{id/product}=="001f"
    ATTRS{id/vendor}=="045e"
    ATTRS{id/version}=="0000"
    ATTRS{inhibited}=="0"
    ATTRS{name}=="IPTS 045E:001F Touchscreen"
    ATTRS{phys}==""
    ATTRS{power/async}=="disabled"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_kids}=="0"
    ATTRS{power/runtime_active_time}=="0"
    ATTRS{power/runtime_enabled}=="disabled"
    ATTRS{power/runtime_status}=="unsupported"
    ATTRS{power/runtime_suspended_time}=="0"
    ATTRS{power/runtime_usage}=="0"
    ATTRS{properties}=="0"
    ATTRS{uniq}==""

  looking at parent device '/devices/pci0000:00/0000:00:16.4/0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04/0000:045E:001F.0003':
    KERNELS=="0000:045E:001F.0003"
    SUBSYSTEMS=="hid"
    DRIVERS=="hid-generic"
    ATTRS{country}=="00"
    ATTRS{power/async}=="enabled"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_kids}=="0"
    ATTRS{power/runtime_active_time}=="0"
    ATTRS{power/runtime_enabled}=="disabled"
    ATTRS{power/runtime_status}=="unsupported"
    ATTRS{power/runtime_suspended_time}=="0"
    ATTRS{power/runtime_usage}=="0"

  looking at parent device '/devices/pci0000:00/0000:00:16.4/0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04':
    KERNELS=="0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04"
    SUBSYSTEMS=="mei"
    DRIVERS=="ipts"
    ATTRS{fixed}=="12"
    ATTRS{max_conn}=="0"
    ATTRS{max_len}=="324"
    ATTRS{name}==""
    ATTRS{power/async}=="disabled"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_kids}=="0"
    ATTRS{power/runtime_active_time}=="0"
    ATTRS{power/runtime_enabled}=="disabled"
    ATTRS{power/runtime_status}=="unsupported"
    ATTRS{power/runtime_suspended_time}=="0"
    ATTRS{power/runtime_usage}=="0"
    ATTRS{uuid}=="3e8d0870-271a-4208-8eb5-9acb9402ae04"
    ATTRS{version}=="0F"
    ATTRS{vtag}=="0"

  looking at parent device '/devices/pci0000:00/0000:00:16.4':
    KERNELS=="0000:00:16.4"
    SUBSYSTEMS=="pci"
    DRIVERS=="mei_me"
    ATTRS{ari_enabled}=="0"
    ATTRS{broken_parity_status}=="0"
    ATTRS{class}=="0x078000"
    ATTRS{consistent_dma_mask_bits}=="64"
    ATTRS{d3cold_allowed}=="1"
    ATTRS{device}=="0x9d3e"
    ATTRS{dma_mask_bits}=="64"
    ATTRS{driver_override}=="(null)"
    ATTRS{enable}=="1"
    ATTRS{irq}=="135"
    ATTRS{local_cpulist}=="0-3"
    ATTRS{local_cpus}=="f"
    ATTRS{msi_bus}=="1"
    ATTRS{msi_irqs/135}=="msi"
    ATTRS{numa_node}=="-1"
    ATTRS{power/async}=="enabled"
    ATTRS{power/autosuspend_delay_ms}=="500"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_kids}=="0"
    ATTRS{power/runtime_active_time}=="28841"
    ATTRS{power/runtime_enabled}=="enabled"
    ATTRS{power/runtime_status}=="suspended"
    ATTRS{power/runtime_suspended_time}=="304297"
    ATTRS{power/runtime_usage}=="0"
    ATTRS{power/wakeup}=="disabled"
    ATTRS{power/wakeup_abort_count}==""
    ATTRS{power/wakeup_active}==""
    ATTRS{power/wakeup_active_count}==""
    ATTRS{power/wakeup_count}==""
    ATTRS{power/wakeup_expire_count}==""
    ATTRS{power/wakeup_last_time_ms}==""
    ATTRS{power/wakeup_max_time_ms}==""
    ATTRS{power/wakeup_total_time_ms}==""
    ATTRS{power_state}=="D0"
    ATTRS{revision}=="0x21"
    ATTRS{subsystem_device}=="0x0000"
    ATTRS{subsystem_vendor}=="0x0000"
    ATTRS{vendor}=="0x8086"

  looking at parent device '/devices/pci0000:00':
    KERNELS=="pci0000:00"
    SUBSYSTEMS==""
    DRIVERS==""
    ATTRS{power/async}=="enabled"
    ATTRS{power/control}=="auto"
    ATTRS{power/runtime_active_kids}=="10"
    ATTRS{power/runtime_active_time}=="0"
    ATTRS{power/runtime_enabled}=="disabled"
    ATTRS{power/runtime_status}=="unsupported"
    ATTRS{power/runtime_suspended_time}=="0"
    ATTRS{power/runtime_usage}=="0"
    ATTRS{waiting_for_supplier}=="0"

Please note the IPTS 045E:001F Touchscreen for event36 and the driver ipts for the device /devices/pci0000:00/0000:00:16.4/0000:00:16.4-3e8d0870-271a-4208-8eb5-9acb9402ae04'.

To me, it seems, that there is still some ipts related driver running, but from here, I do not know, how to further investigate.

But maybe this gives a hint, what still enables the touchscreen, although I expected that I should not be able to do any touch input after purging iptsd?

[StollD]

But even after several reboots and shutdowns after executing the commands above, the touchinput for fingers still work, meaning I can use my fingers with the touchscreen (but not the surface pen). Doesn`t this contradict to what you said earlier?:

The hardware provides two different modes: One that sends raw multitouch data that needs processing, and a fallback mode that sends singletouch data as ready to use coordinates.

The singletouch fallback is handled directly in the kernel driver. When iptsd is started, it switches the driver into the multitouch mode, transforms the raw data from the touch sensor into usable coordinates, and then emits them through its own input devices. If iptsd isn't running you shouldnt be able to zoom in Firefox, because only a single finger will be recognized.

[StollD]

This issue on linux-surface is very similar if not identical to your issue: https://github.com/linux-surface/linux-surface/issues/1199

They are using a Surface Go which is supported through an upstream driver, so this very much sounds like an Ubuntu / Gnome regression.

[ArPhil]

Thanks again for the hints.

I took the time to reinstall everything on my surface, since being able to use the surface pen in portrait mode is an essential feature for me.

The result:

• After installing a fresh Ubuntu-22.04 LTS, I followed the instructions here and executedsudo apt install linux-image-surface linux-headers-surface libwacom-surface iptsdand rebooted. Everything worked fine then. That includes the orientation of the pencil in portrait mode.
• After executing sudo apt update && sudo apt upgrade and rebooting / restarting, pencil still worked fine. So I was happy (at first.)
• I noticed, that after the upgrade, some packages were not updated / were kept back. The list of packages is

  gjs/jammy-updates 1.72.4-0ubuntu0.22.04.1 amd64 [upgradable from: 1.72.2-0ubuntu1]
  libgjs0g/jammy-updates 1.72.4-0ubuntu0.22.04.1 amd64 [upgradable from: 1.72.2-0ubuntu1]
  python3-distupgrade/jammy-updates,jammy-updates 1:22.04.17 all [upgradable from: 1:22.04.16]
  ubuntu-release-upgrader-core/jammy-updates,jammy-updates 1:22.04.17 all [upgradable from: 1:22.04.16]
  ubuntu-release-upgrader-gtk/jammy-updates,jammy-updates 1:22.04.17 all [upgradable from: 1:22.04.16]

• I installed gjs and the other packages manually (do not remember the installation order or whether I used apt full-upgrade or apt-get dist-upgrade and rebooted). At the end, I ended up with the same error regarding pencil orientation in portrait mode as mentioned above. I was not able to uninstall the packages and restore the correct behaviour of the pencil.
• I performed another clean installation of ubuntu and left the packages mentioned above untouched. So far everything works fine for me, and I hope, that these packages will be kept back. I could mark them as not upgradable as well.

Some hints:

• I am not sure, whether it was me, or the normal update process, but somehow the iptsd stuff got compromised, although I did not any system changes. My setup was untouched for quite a while now. Only thing I did was updates from the provided surface stream.
• The wacom tab in settings, still does not show a detected pencil. I guess this is the expected behaviour, but still confusing.
• Also it detects my tablet as surface pro 6. I know, that it was said, that microsoft reused components / boards and I would understand, when my surface pro 6 was detected as surface pro 5, but not vice versa. However maybe this still is irrelevant.

A suggestion / feature request:

• Not sure if this makes sense, but maybe the config for iptsd and iptsd itself could be extended with the option to override the behaviour / interpretation of touch / pencil events with respect to orientation. Meaning, I would like to be able to override in an iptsd config:

1. x-axis orientation (for touch events and pencil) for landscape mode
2. y-axis orientation (for touch events and pencil) for landscape mode
3. x-axis orientation (for touch events and pencil) for portrait mode
4. y-axis orientation (for touch events and pencil) for portrait mode

I understand, that when wrong coordinates / interpretation of touch events is sent to iptsd in the user space, it is rather an error "below" and not within iptsd itself. And I also understand, that transforming the received coordinates could affect performance (cannot estimate how much), but maybe this could help in cases, where the layer below cannot be fixed. So this would enable correcting any "wrong" touch input at iptsd level.

I am curious to know, what you @StollD , do think about this.

In any case thanks a lot for your investigation time, since this revealed, that the error regarding pencil orientation is not within iptsd!!! I am good for now and you can close the isse, unless you further want to comment on this. :)

Thank!!!

[StollD]

Nice work! Although I honestly have no idea why these packages would trigger it. The ubuntu and python packages are probably fine but gjs is from the GNOME project.

• The wacom tab in settings, still does not show a detected pencil. I guess this is the expected behaviour, but still confusing.

I feel like this shouldnt happen, but its been a while since I used GNOME and maybe it is related to the older version that Ubuntu uses.

• Also it detects my tablet as surface pro 6. I know, that it was said, that microsoft reused components / boards and I would understand, when my surface pro 6 was detected as surface pro 5, but not vice versa. However maybe this still is irrelevant.

Well there are two different Surface Pro 5: The normal one and the one with LTE. The normal one has its own device ID on the digitizer, the LTE version simply shares one with the Pro 6.

We simply got the first report of that ID from a Pro 6, so thats what we named it in libwacom.

• https://github.com/linux-surface/libwacom/blob/6b9a7b050064e7300c72f374d5667afdffca04de/data/surface-pro5.tablet#L6
• https://github.com/linux-surface/libwacom/blob/6b9a7b050064e7300c72f374d5667afdffca04de/data/surface-pro6.tablet#L6

I understand, that when wrong coordinates / interpretation of touch events is sent to iptsd in the user space, it is rather an arrow "below" and not within iptsd itself. And I also understand, that transforming the received coordinates could affect performance (cannot estimate how much), but maybe this could help in cases, where the layer below cannot be fixed. So this would enable correcting any "wrong" touch input at iptsd level.

In principle that functionality already exists because the hardware sometimes sends mirrored data (I think when they installed the digitizer the wrong way or something like that). Its just applied to both touch and stylus.

I could add seperate options for both, but it feels like a terrible hack and will lead to people using it instead of getting issues fixed upstream. And it also won't help the Surface Go users that don't use iptsd.

[ArPhil]

The ubuntu and python packages are probably fine but gjs is from the GNOME project.

Yes, that was my first thought too. :)

Well there are two different Surface Pro 5: The normal one and the one with LTE. The normal one has its own device ID on the digitizer, the LTE version simply shares one with the Pro 6.

That clearifies now everything for me... I have the LTE version. Sorry for bothering that much regarding this.

I could add seperate options for both, but it feels like a terrible hack and will lead to people using it instead of getting issues fixed upstream. And it also won't help the Surface Go users that don't use iptsd.

I totally agree, that it is a "hack", also I did not consider, that this would not work for go users. However, what I consider too is, that even when the origin of the error is not within iptsd, I believe, that it would help a lot having the outlined options fixing the orientation with a config file. I mean, this issue here has proven, that we know, that the error is not within iptsd, but we also were not able to locate and fix it entirely. So the workaround (for me) was to reinstall everything, which definitely took more time, than it would have taken by setting an overriding config option. Also in case of similar errors but with other origins than one of the packages identified above, could be fixed by users rather easily. I do not consider myself an expert regarding Linux and package management but I believe, that there are some linux surface users, that really just follow the instructions provided in the documentation and maybe are not able to investigate by themselves, especially since this requires deeper understanding of the (gnome) architecture and packages.

But these are just thoughts, and I still understand the / your point, that it feels like a hack. I would agree, but I believe, that I see the benefits in having an option in the config as well. :)

[ArPhil]

One last update:

The packages I mentioned a few comments above (here) do NOT seem to be the issue.

I installed them (including gjs) via the Ubuntu Software Updater (not using apt / apt-get in cli) and it seems to work fine, even after several reboots and restarts.

However, as I mentioned earlier: during my first reinstallation of ubuntu, I also managed to break things by installing (some or all of) the listed packages via cli using apt / apt-get. Not sure what is different than using the ubuntu software updater (expected the gui application to rely on apt / apt-get) or whether this was just some strange coincident.

So currently I have all newest package updates installed and neither apt / apt-get nor the software updater do suggest any further package updates. That brings me to the conclusion, that there (probably) is not a regression upstream, but that rather during updating (some) packages can break the proper functioning of the pencil in portrait mode.

I have no idea how to avoid having the issue in the future. Maybe do backups of the system regularly and rollback in case the pencil stops working properly.