Hardware buttons on SP9 not working anymore after updating to arch-6.7.9-1

FearlessSpiff commented 2 months ago

Hi

I just updated to arch-6.7.9-1 and now my hardware buttons (PWR, Vol+ and Vol-) aren't working anymore. So I can't set the SP9 to sleep with the button and if I do otherwise I can't wake it up anymore!?! What am I missing? Where there changes I should be aware of?

Thanks!

Environment

Hardware model: Surface Pro 9
Kernel version: arch-6.7.9-1
Distribution: Arch Linux

dmesg.log

FearlessSpiff commented 2 months ago

Guess this is wrong alarm, but I don't understand it. I now downgraded the kernel but still the same issue. Any idea which package I updated could have caused this

[2024-03-11T18:38:13+0100] [ALPM] upgraded atuin (18.0.2-1 -> 18.1.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded xz (5.6.0-1 -> 5.6.1-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded bluez (5.72-2 -> 5.73-3)
[2024-03-11T18:38:13+0100] [ALPM] upgraded bluez-libs (5.72-2 -> 5.73-3)
[2024-03-11T18:38:13+0100] [ALPM] upgraded bluez-utils (5.72-2 -> 5.73-3)
[2024-03-11T18:38:13+0100] [ALPM] upgraded brltty (6.6-5 -> 6.6-6)
[2024-03-11T18:38:13+0100] [ALPM-SCRIPTLET] Please add your user to the brlapi group.
[2024-03-11T18:38:13+0100] [ALPM] upgraded cairo (1.18.0-1 -> 1.18.0-2)
[2024-03-11T18:38:13+0100] [ALPM] upgraded knewstuff (6.0.0-3 -> 6.0.0-4)
[2024-03-11T18:38:13+0100] [ALPM] upgraded poppler (24.01.0-1 -> 24.03.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded poppler-qt6 (24.01.0-1 -> 24.03.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded libjxl (0.10.1-1 -> 0.10.2-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded librsvg (2:2.57.1-1 -> 2:2.57.2-1)
[2024-03-11T18:38:13+0100] [ALPM] installed qt6-shadertools (6.6.2-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded qt6-5compat (6.6.2-1 -> 6.6.2-2)
[2024-03-11T18:38:13+0100] [ALPM] upgraded libplist (2.3.0-2 -> 2.4.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded dolphin (24.02.0-1 -> 24.02.0-2)
[2024-03-11T18:38:13+0100] [ALPM] upgraded gcr-4 (4.2.1-1 -> 4.2.1-2)
[2024-03-11T18:38:13+0100] [ALPM] upgraded ghostscript (10.02.1-1 -> 10.03.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded gnome-disk-utility (45.1-1 -> 46.0-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded intel-media-driver (24.1.3-1 -> 24.1.4-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded just (1.24.0-1 -> 1.25.2-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded libfabric (1.20.0-1 -> 1.20.1-1)
[2024-03-11T18:38:13+0100] [ALPM] upgraded libnbd (1.18.2-1 -> 1.18.2-2)
[2024-03-11T18:38:13+0100] [ALPM] upgraded libxaw (1.0.15-1 -> 1.0.16-1)
[2024-03-11T18:38:14+0100] [ALPM] upgraded linux-surface (6.7.6.arch1-1 -> 6.7.9.arch1-1)
[2024-03-11T18:38:14+0100] [ALPM] upgraded poppler-glib (24.01.0-1 -> 24.03.0-1)
[2024-03-11T18:38:14+0100] [ALPM] upgraded poppler-qt5 (24.01.0-1 -> 24.03.0-1)
[2024-03-11T18:38:14+0100] [ALPM] upgraded tree-sitter (0.20.10-2 -> 0.22.1-2)
[2024-03-11T18:38:14+0100] [ALPM] upgraded wget (1.21.4-1 -> 1.24.5-1)
[2024-03-11T18:38:14+0100] [ALPM] upgraded zziplib (0.13.73-1 -> 0.13.74-1)

FearlessSpiff commented 2 months ago

Any hint how I can look for hints about the buttons? Tried wev but nothing shows up. Also libinput list-devices and I think the buttons are missing

21:02:04 ~ $ sudo libinput list-devices
Device:           Video Bus
Kernel:           /dev/input/event11
Group:            1
Seat:             seat0, default
Capabilities:     keyboard 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           Lid Switch
Kernel:           /dev/input/event0
Group:            2
Seat:             seat0, default
Capabilities:     switch
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           Intel Touch Host Controller Touchscreen
Kernel:           /dev/input/event6
Group:            3
Seat:             seat0, default
Size:             273x181mm
Capabilities:     touch 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           Intel Touch Host Controller Stylus
Kernel:           /dev/input/event9
Group:            3
Seat:             seat0, default
Size:             274x185mm
Capabilities:     tablet 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      disabled
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   none
Rotation:         n/a

Device:           Microsoft Surface POS Tablet Mode Switch
Kernel:           /dev/input/event2
Group:            4
Seat:             seat0, default
Capabilities:     switch
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      n/a
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           IPTS Touch
Kernel:           /dev/input/event16
Group:            5
Seat:             seat0, default
Size:             274x185mm
Capabilities:     touch 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      n/a
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   n/a
Rotation:         0.0

Device:           IPTS Stylus
Kernel:           /dev/input/event17
Group:            6
Seat:             seat0, default
Size:             274x185mm
Capabilities:     tablet 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      disabled
Nat.scrolling:    n/a
Middle emulation: n/a
Calibration:      identity matrix
Scroll methods:   none
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   none
Rotation:         n/a

Device:           Logitech M720 Triathlon Multi-Device Mouse
Kernel:           /dev/input/event19
Group:            7
Seat:             seat0, default
Capabilities:     keyboard pointer 
Tap-to-click:     n/a
Tap-and-drag:     n/a
Tap drag lock:    n/a
Left-handed:      disabled
Nat.scrolling:    disabled
Middle emulation: disabled
Calibration:      n/a
Scroll methods:   button
Click methods:    none
Disable-w-typing: n/a
Disable-w-trackpointing: n/a
Accel profiles:   flat *adaptive custom
Rotation:         0.0

FearlessSpiff commented 2 months ago

Ok, saw that I upgraded from 6.7.2.arch1-1 on Friday and most likely did not reboot. Now it works again, when downgrading to 6.7.2.arch1-1. So this sure is a regression?!? Is this the right place to mention this or is it Arch related? Here my upgrade history with comments

[2024-03-08T18:17:59+0100] [ALPM] upgraded linux-surface (6.7.2.arch1-1 -> 6.7.6.arch1-1)   # -> Problem started here
[2024-03-11T18:38:14+0100] [ALPM] upgraded linux-surface (6.7.6.arch1-1 -> 6.7.9.arch1-1)   # -> Buttons DEAD

MeisterLLD commented 2 months ago

Same problem here on Surface Pro 2017 with 6.7.6-Arch

MeisterLLD commented 2 months ago

Problem persists after update to Linux archsurface 6.7.9-arch1-1-surface #1 SMP PREEMPT_DYNAMIC Sat, 09 Mar 2024 20:28:40 +0000 x86_64 GNU/Linux

FearlessSpiff commented 2 months ago

Bad for us, but good to see that I'm not alone...

MeisterLLD commented 2 months ago

Downgrade to 6.7.2 solved it too, so I guess the problem appeared at 6.7.6

dev-nis commented 2 months ago

Downgrade to 6.7.2 solved it too, so I guess the problem appeared at 6.7.6

I'm on the Debian/Ubuntu-release of the Surface-Linux-Kernel, currently on version 6.7.6-surface-1 and I don't face this issue.

drguell commented 2 months ago

I can confirm the same issue on Surface 7 pro+ with Archlinux:

Linux surface 6.7.9-arch1-1-surface #1 SMP PREEMPT_DYNAMIC Sat, 09 Mar 2024 20:28:40 +0000 x86_64 GNU/Linux

If you need more information, tell me ;)

StollD commented 2 months ago

The buttons have a bit of a picky setup, so I will try my best to explain this:

You need to consider two drivers: pinctrl and soc_button_array.

pinctrl is not actually a driver but more a family of drivers. Every platform has their own pinctrl driver. For example, my SB2 is pinctrl_sunrisepoint and SP9 is probably pinctrl_alderlake. These drivers power the GPIO pins that the buttons are connected to.

soc_button_array is the driver that actually powers the buttons through GPIO. When it is loaded, it expects that pinctrl has already set up the GPIO interface. Except that the structure of one separate pinctrl driver per platform makes it impossible to actually express that dependency.

The actual issue at hand is a race condition between both drivers: soc_button_array gets loaded before pinctrl, fails to talk to the GPIO device and fails.

There are two ways to solve this issue:

Build all pinctrl drivers into the kernel and not as modules.
Build all pinctrl drivers as modules, but add them to the initramfs to force the kernel to load them before it can access soc_button_array from the root file system.

Before 6.7.4 Arch used the first option and had all pinctrl drivers in vmlinux. Now it builds them as modules but doesn't add them to the initramfs, triggering the race condition.

This also only affects Arch. Fedora puts pinctrl into the initramfs, and I think Debian has them built-in.

To solve this right now you have again two options:

Run sudo modprobe -r soc_button_array && sudo modprobe soc_button_array after every reboot
Run lsmod | grep pinctrl and add everything it outputs to your initramfs

Please also report this to upstream Arch, because this is not surface specific.

I can add a downstream patch to our builds that puts pinctrl into vmlinux again, but I would really prefer if Arch would fix this themselves, because pinctrl is like 20 different drivers and keeping up with that will be annoying.

FearlessSpiff commented 2 months ago

Thanks a lot. I will try the initramfs option tomorrow. And will check to report upstream Arch. Have a good night!

drguell commented 2 months ago

Super explanation, @StollD. Thanks for it.

I tried the two options and them work like a charm.

I used the following instructions to add my pinctrl module to initramfs:

List my module (as said @StollD )

# lsmod | grep pinctrl
pinctrl_tigerlake      28672  13

Add module into initramfs:

/etc/mkinitcpio.conf

MODULES=(pinctrl_tigerlake)
BINARIES=()
FILES=()
HOOKS=(base udev autodetect keyboard keymap modconf block filesystems fsck)

Rebuild all of initramfs images:
```
# mkinitcpio -P 
```

I hope this help someone.

Leo1998 commented 2 months ago

Solution worked thanks alot!! Could this maybe be related to the SAM crash from #1319? I noticed that removing and then modprobing the surface_aggregator module brings back the battery status.

JackYansongLi commented 2 months ago

same issue on sp6

qzed commented 2 months ago

Solution worked thanks alot!! Could this maybe be related to the SAM crash from #1319? I noticed that removing and then modprobing the surface_aggregator module brings back the battery status.

@Leo1998 Might be related, yes. Could you check if this fixes that as well and if it does let the others know about it on #1319 as well?

zash1958 commented 2 months ago

Works also here and the hardware buttons work again as expected. Device SP5 You have to change the mkinitcpio to load

pinctrl_sunrisepoint

instead of the module _pinctrltigerlake and all is fine now

Leo1998 commented 2 months ago

Solution worked thanks alot!! Could this maybe be related to the SAM crash from #1319? I noticed that removing and then modprobing the surface_aggregator module brings back the battery status.

@Leo1998 Might be related, yes. Could you check if this fixes that as well and if it does let the others know about it on #1319 as well?

@qzed Unfortunately, this did not fix the surface aggregator crash :(

jurica commented 1 month ago

Thanks a lot. I will try the initramfs option tomorrow. And will check to report upstream Arch. Have a good night!

May I ask where you have reported this to Arch? I'd like to keep an eye on the report and see if and how Arch handles it.

FearlessSpiff commented 1 month ago

Thanks a lot. I will try the initramfs option tomorrow. And will check to report upstream Arch. Have a good night!

May I ask where you have reported this to Arch? I'd like to keep an eye on the report and see if and how Arch handles it.

Oops. I forgot it. Typical. Once it works again, I forget the stuff. :-(

Maybe somebody else did?

linux-surface / linux-surface

Hardware buttons on SP9 not working anymore after updating to arch-6.7.9-1 #1392

Environment