Lenovo x230t : BackSpace keycode (c:160) is mapped to XF86ScreenSaver, not BackSpace

tlaurion commented 6 years ago

Qubes OS version:

R3.2, R4.0rc1, R4.0rc2

Affected TemplateVMs:

dom0 fedora-23, dom0 fedora-25

Steps to reproduce the behavior:

Install Qubes 3.2. At login prompt, backspace doesn't do its backspace function anymore.

Expected behavior:

Under x230t (not true for x230!?) doing sudo xev key shows that it's mapped to BackSpace

Actual behavior:

sudo xev and typing the backspace key shows that it's linked to XF86ScreenSaver, not BackSpace

General notes:

Corrective action is to add the following line in the file /etc/X11/Xmodmap: keycode 160 = BackSpace NoSymbol BackSpace

Related issues:

tlaurion commented 6 years ago

Doesn't resolve the issue after reboot. Any hint?

marmarek commented 6 years ago

I guess /etc/X11/Xmodmap isn't loaded by lightdm...

marmarek commented 6 years ago

Does it work correctly on plain Fedora 25 (or newer)?

tlaurion commented 6 years ago

Overwriting /etc/X11/Xmodmap with the following file makes the change persistent over 4.0 rc3, effective from dom0 and launched qubes. Didn't verify for 3.2.

xmodmap_x230t.txt

tlaurion commented 6 years ago

My x230t is dead. Cannot test this but for anyone else with the same kind of problem:

forums.puri.sm/t/keyboard-layout-unable-to-recognize-pipe/2022

Reference

Anyone with a x230t that could give the content of /sys/class/dmi/id/modalias ? I suspect it should differenciate x230t of x230. If it is, it would be possible to deploy a permanent fix.

tlaurion commented 6 years ago

X230t resuscitated.

modalias content: dmi:bvncoreboot:bvrCBET4000heads:bd01/01/1970:svnLENOVO:pn343727U:pvrThinkPadX230Tablet:rvnLENOVO:rn343727U:rvrThinkPadX230Tablet:cvnLENOVO:ct9:cvr:

andrewdavidwong commented 3 years ago

This issue is being closed because:

This issue has the "Release 3.2 updates" milestone.
Qubes OS 3.2 has reached end-of-life (EOL).
This issue has not been updated in over a year.

If anyone believes that this issue should be reopened, please let us know in a comment here.

matthewspangler commented 1 year ago

I can confirm I am having this issue on the following setup:

Thinkpad X230T Qubes OS 4.1.2 Heads-x230-hotp-maximized-v0.2.0-1577-g91f65be

https://app.circleci.com/pipelines/github/osresearch/heads/612/workflows/87b50d50-6afd-46bc-9130-7475bb9fc1ee/jobs/10337/artifacts

Both the 'Backspace' and | \ keys are unresponsive.

I have tried the following fix detailed in these two links:

https://github.com/osresearch/heads/issues/958#issuecomment-1530639642
systemd/systemd@3bbdb32#diff-df937b870825d805803c76ae8258342dcd6850770f4404721e044af93ac3d231R963

Here are the steps for this fix: 1) Edit /usr/lib/udev/hwdb.d/60-keyboard.hwdb in dom0 2) Find the line containing: Thinkpad*X2*Tablet* (for me it was around line 963) 3) Add :rvn to the end of the line, like so: `evdev:atkbd:dmi:bvn:bvr:bd:svnLENOVO:pn:pvrThinkPadX2T:rvn`

Afterwards both keys are working, but occasionally during daily usage I'll find that the issue pops up again. I'm still trying to replicate the conditions where the fix regresses.

EDIT 11/16/23: I doubt the above fix actually worked, it may have just been because I was warm booting and it appeared to resolve.

matthewspangler commented 9 months ago

I've figured out why this bug sometimes appears for me, and sometimes doesn't.

Bug appears when: 1) Do a cold boot. Unplug battery, unplug power cable, hold power button to drain power -- then boot. 2) Type LUKS password -- at this point, backspace and "| \" work in the LUKS text box 3) User login -- at this point, backspace and "| \" have stopped working, and continue to not work after login.

Bug doesn't appear when: 1) Do warm boot by rebooting from Qubes OS. 2) Keys work fine @ LUKS prompt + user login + desktop.

This is talked about in this Heads issue: https://github.com/linuxboot/heads/pull/1525#issuecomment-1810802483

The issue seems to be related to i8042.c in the linux kernel per https://github.com/linuxboot/heads/issues/958#issuecomment-1533908663

toothlesslizard commented 9 months ago

Same on Qubes 4.2 and x230 from xytech (keyboard from x220). I have Ubuntu 23.10 on second disk drive (nvme mod + ssd) and there is no issues.

Qubes 4.2.0-rc4 xen 4.17.2 Kernel 6.5.8-1 dom0: fedora-37 FW_VER - Heads-v0.2.0-1796-g35f3b22-dirty Kernel: Linux 5.10.5 - heads X230-maximized-eDP

tlaurion commented 9 months ago

Somebody reported that using q4.2 changed the behavior. Went extensively with the user and this happens only on cold boot, where reboot detect the keyboard correctly.

Curated the logs and opened a PR under heads at https://github.com/linuxboot/heads/pull/1525 which contains his logs.

The bug is that on cold boot, i8042 probes the keyboard and detects it as Raw translated 2, where on reboot it's detected correctly as AT translated

Unfortunately I do not understand what is happening there. But the logs have i8042 in debug mode with kernel option to not timeout which is the best I could come up to gather logs from the user.

No clue what is happening there and cannot replicate on my x230. This is not a x230t here. Keyboard sku still unknown.

On Thu, Nov 16, 2023, 3:28 PM toothlesslizard @.***> wrote:

Same on Qubes 4.2 and x230 from xytech (keyboard from x220). backspace, /, | this keys work intermittently I have Ubuntu on second disk drive and there is no troubles.

Qubes 4.2 FW_VER - Heads-v0.2.0-1796-g35f3b22-dirty Kernel: Linux 5.10.5 - heads X230-maximized-eDP

— Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-1815264617, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGKBMURLFFFERSHWQB5IBDYEZZQRAVCNFSM4EDQQP4KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBRGUZDMNBWGE3Q . You are receiving this because you modified the open/close state.Message ID: @.***>

toothlesslizard commented 9 months ago

@tlaurion do you need dmesg +sku? this bug not only 230t related. Try x230 + 45n2141 (for example) (maybe ec-firmware? but other distros works flawlessly)

tlaurion commented 9 months ago

I wonder if that would fix it under x230 under coreboot

--- a/src/drivers/i8042/i8042.c
+++ b/src/drivers/i8042/i8042.c
@@ -132,7 +132,11 @@ static int i8042_init(void)
    /* Enable keyboard interface */
    i8042_command(&param, I8042_CMD_CTL_WCTR);

+#if CONFIG_BOARD_LENOVO_X230
+   param |= I8042_XLATE;
+#else
    param &= ~I8042_XLATE;
+#endif
    i8042_command(&param, I8042_CMD_CTL_WCTR);

    /* Flush any pending input. */

Problem is

--- a/src/drivers/i8042/i8042.h
+++ b/src/drivers/i8042/i8042.h
@@ -23,7 +23,7 @@
 #define I8042_AUX_OBF      0x20    /* output buffer full */
 #define I8042_AUX_IRQ      12

-#define I8042_XLATE        0x00    /* translate scan code set 2 to 1 */
+#define I8042_XLATE        0x40    /* translate scan code set 2 to 1 */
 #define I8042_MUX_QUIRK        0x02    /* enable multiplexing quirk */

 #define I8042_KBD_IRQ      1

matthewspangler commented 9 months ago

I'm ignorant as to a lot of this, but I'm going to take a wild guess--what about kernel option "i8042.probe_defer" in kernels >=6.7? It's related to integrated PS/2 keyboards not being ready at cold boot because they're busy performing a self-test.

https://github.com/torvalds/linux/commit/9222ba68c3f4065f6364b99cc641b6b019ef2d42 https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.17/+bug/76881

tlaurion commented 9 months ago

Analysis of debug logs at https://github.com/linuxboot/heads/pull/1525#issuecomment-1816501783

tlaurion commented 9 months ago

EDIT nevermind, past comment misleading.

tlaurion commented 9 months ago

I'm ignorant as to a lot of this, but I'm going to take a wild guess--what about kernel option "i8042.probe_defer" in kernels >=6.7? It's related to integrated PS/2 keyboards not being ready at cold boot because they're busy performing a self-test.

https://github.com/torvalds/linux/commit/9222ba68c3f4065f6364b99cc641b6b019ef2d42 https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.17/+bug/76881

I think this would happen too late. My reasoning to push for a detection fix or hardcoding on coreboot side is that Linux should do the right thing if coreboot result was good where Heads Linux just uses the keyboard as detected without udev fixes. If detection was stable there, workarounds would not be needed later on. At least that's my reasoning which could be faulty.

After that, os apply or not their udev fix, as seen in this thread. If os applies udev fix whatever raw/at then behavior might still be unexpected unless i8042 initiates a reset on controller.

tlaurion commented 9 months ago

I think I fixed it with ugly workaround, disabling exits on pc80 code of coreboot. Needs testing to raise attention from coreboot developers.

Should work for x220t/x230t/x230.

See https://github.com/linuxboot/heads/pull/1525#issuecomment-1817058361

Edit: it "worked". From Heads recovery shell with that rom tested, coldboot:

But coreboot says that pc80 driver for ps2 is not used and that it adds delay and that the payload should handle with ps2 init and keyboard proper init.

Meaning we have to find the proper i8042 kernel parameters to fixate At translated 2 when coreboot boots Heads first kernel.

That means disabling ps2 support from coreboot and having coreboot config pass to the kernel the right i8042 drivers parameters to permanently fix the issue.

Someone having misbehaving hardware can play around and propose a PR based on the commits of testing PR?

tlaurion commented 9 months ago

Looking at i8042 source, it seems that passing from coreboot config to heads kernel : CONFIG_LINUX_COMMAND_LINE="i8042.force=1:kbd" should actually be enough

(or force this in grub config for easier qubesos testing ).

Anyone can test and report back?

toothlesslizard commented 9 months ago

strange, problem persists. added these lines and rebuild rom. lightdm greeter won't respond to the backspace key. or iam doing something wrong?

user@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep 8042
CONFIG_LINUX_COMMAND_LINE="intel_iommu=igfx_off i8042.force=1:kbd quiet"

Qubes grub no effect too.

tlaurion commented 9 months ago

strange, problem persists. added these lines and rebuild rom. lightdm greeter won't respond to the backspace key. or iam doing something wrong?
user@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep 8042
CONFIG_LINUX_COMMAND_LINE="intel_iommu=igfx_off i8042.force=1:kbd quiet"
Qubes grub no effect too.

Well. That doesn't seem enough. Can you confirm from dmesg output of 'sudo dmesg | grep -i keyb` under heads recovery shell and final os and 'cbmem -1 | grep -I keyb' under heads recovery shell please.

tlaurion commented 9 months ago

@toothlesslizard also if you could deactivate CONFIG_DRIVERS_PS2_KEYBOARD from coreboot config that would make a consistent test without coreboot ps2 init being in the way whatsoever.

behavior should be different between cold boot (power off, holding power 10 seconds) and warm boot (reset, reboot) as for everybody else.

toothlesslizard commented 9 months ago

@tlaurion I'm getting different outputs from the shell recovery (on/off laptop) :

[          0.828917] input : AT Translated Set 2 keyaboard as /devices/platform/i8042/serio0/input/input3

[DEBUG] Keyboard init ...
[ INFO ] Keyboard controller output buffer result timeout
[DEBUG] PS/2 Keyabord initialized on primary channel

or this

[     0.824340] input : AT RAW Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[     1.692471] input : AT RAW Set 2 keyboard as /devices/platform/i8042/serio0/input/input6

[DEBUG] Keyboard init ...
[ INFO ] Keyboard controller output buffer result timeout
[ INFO ] Keyboard controller output buffer result timeout
[ INFO ] Keyboard controller output buffer result timeout
[ERROR] Keyboard reset failed ASK: 0xaa

rebuilding the rom from scratch for now again.

# CONFIG_DRIVERS_PS2_KEYBOARD is not set

toothlesslizard commented 9 months ago

full output from Qubes OS. cant filtering cause keys not working. full-dmesg.txt

new rom from scratch CBET4000 Heads-v0.2.0-1914-g1f39d16-dirty stock bios from here : https://github.com/xy-tech/x330-bios/tree/main/stock/original

user@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep -i "8042"
CONFIG_LINUX_COMMAND_LINE="intel_iommu=igfx_off i8042.force=1:kbd quiet loglevel=2"
user@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep -i "KEYBO"
# CONFIG_DRIVERS_PS2_KEYBOARD is not set

cbmem | grep keyb or Keyboa - no outut now. other things about AT Translated / AT RAW same. See prev post.

If anyone interesting i have different sku keyboards. But swapping not help.

Boot from second drive and output from Ubuntu (keys works good)

toothless@ubuntu:~$ sudo dmesg | grep -i Keyb
[sudo] password for toothless: 
[    0.387853] input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[   21.529352] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...

tlaurion commented 9 months ago

new rom from scratch\nCBET4000 Heads-v0.2.0-1914-g1f39d16-dirty\nstock bios from here : https://github.com/xy-tech/x330-bios/tree/main/stock/original\n\nuser@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep -i \"8042\"\nCONFIG_LINUX_COMMAND_LINE=\"intel_iommu=igfx_off i8042.force=1:kbd quiet loglevel=2\"\nuser@heads:~/heads/config$ cat coreboot-x230-maximized-fhd_edp.config | grep -i \"KEYBO\"\n# CONFIG_DRIVERS_PS2_KEYBOARD is not set\ncbmem | grep keyb or Keyboa - no outut now.\nother things about AT Translated / AT RAW same. See prev post.\n\nIf anyone interesting i have different sku keyboards. But swapping not help.

I'm sorry @toothlesslizard i didn't get that part.

Are you saying that on stock bios (x330 referenced) you are getting the same coldboot=raw, reboot/warm reset=at?

toothlesslizard commented 9 months ago

No, I just added details about the firmware build, nothing more. Its about blobs in xx30 folder. The point is, the keys continue to malfunction. Only cold boot. Recovey shell and LUKS - good, LightDM Greeter - keys fall off.

tlaurion commented 9 months ago

Boot from second drive and output from Ubuntu (keys works good)

The Raw Translated 2 being dealt with Ubuntu is nice (systemd workarounds?) but is what differenciate quebesos from others here (where some could apply fixes to qubesos, but those won't stick if behavior is different between coldboot and warm boot and is not fixated across them)

Raw Translated 2 should not happen. We want AT Translated 2, otherwise we have different behaviors across OSes because of workarounds applied differently across OSes. If AT translated 2: no problem across all OSes.

The question is how to arrive to that. With hacks on coreboot disabling exit 0 on patch referred above, we arrive to that state but that is really ugly.

So this is where we are here. No real progress knowing which kernel parameters need to be set so that if firmware doesn't get in the way, we get to AT translated 2, otherwise OSes have to apply quirks of their own. Which won't work if ps2 protocol switches between AT translated 2 to Raw Translated 2 depending if we are cold booting or warm booting. I hope we are now clear on what the problem is if we haven't found proper solution... Yet.

tlaurion commented 9 months ago

Opened issue https://ticket.coreboot.org/issues/515

QubesOS / qubes-issues