search

linuxboot / heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible boot environment for laptops, workstations and servers.
https://osresearch.net/
GNU General Public License v2.0
1.4k stars 181 forks source link

x230: keyboard not working after reboot #765

Closed alex-nitrokey closed 2 years ago

alex-nitrokey commented 4 years ago

I experience regularly an issue that is causing the keyboard or the x230 to not function after reboot inside of heads. Killing heads via pressing the power button until power off and starting again fixes the problem.

This problem is happening often after reboot (again, reboots inside heads), but not always. I could not detect a pattern yet.

Anybody had this issue or can deduce a cause?

paulmenzel commented 4 years ago

Maybe related to https://github.com/osresearch/heads/issues/698?

alex-nitrokey commented 4 years ago

doesn't sound like it imho

stmc-droid commented 4 years ago

I have the same issue sometimes when rebooting inside heads

paulmenzel commented 4 years ago

Are you able to save the logs somewhere even without keyboard?

Please start the Linux (Heads) kernel with i8042.debug, and contact the Linux kernel developers with the failing logs.

tlaurion commented 4 years ago

Intuition here is a race condition with USB Security dongle being connected and considered a keyboard. Possible? Would need to test and isolate if behavior is there without USB Security dongle being connected at boot.

alex-nitrokey commented 4 years ago

Good idea! I try to test this...

alex-nitrokey commented 4 years ago

Can not confirm. I tried around just a bit, but while doing similar stuff like generating new HOTP secrets and reboot via Ctrl+Alt+Del I once got stuck with not working keyboard and once I didn't (in both cases I led the Nitrokey inserted).

tlaurion commented 4 years ago

@alex-nitrokey best point would be to see if the same behavior happens without the Nitrokey Inserted and why the Nitrokey is considered a USB keyboard in Heads use case. (new behavior)

alex-nitrokey commented 4 years ago

I don't think, it has anything to do with the Nitrokey, this was just a working hypothesis above.

I just tested rebooting without inserting the Nitrokey or any other USB device. The keyboard still freezes.

alex-nitrokey commented 4 years ago

(go to recovery shell without doing anything else and type reboot)

tlaurion commented 4 years ago

Just putting some notes, I will do some Heads pull request merging in the next days after testing them under a branch under CIs.

On current issue: reboot under heads is a bash script:

user@x230-master:~/heads$ cat initrd/bin/reboot 
#!/bin/sh

# Sync all mounted filesystems
echo s > /proc/sysrq-trigger

# Remount all mounted filesystems in read-only mode
echo u > /proc/sysrq-trigger

# Immediately reboot the system, without unmounting or syncing filesystems
echo b > /proc/sysrq-trigger

Please detail your use case: 1- Call reboot inside of heads. 2- Have main Heads screen but not able to type anything even if no USB drive being present on rebooted system? Is that correct?

@alex-nitrokey :

Please start the Linux (Heads) kernel with i8042.debug, and contact the Linux kernel developers with the failing logs. That means putting that under coreboot config. Result?

alex-nitrokey commented 4 years ago

Please detail your use case: 1- Call reboot inside of heads. 2- Have main Heads screen but not able to type anything even if no USB drive being present on rebooted system? Is that correct?

This is one test case, yes. But it happens in different reboot situations as well, like doing the oem-factory-reset (in which I of course needed to insert USB devices). The above use case is just proving that seemingly it has nothing to do with the USB drive as it is happening even without inserting any.

Will try next week with the kernel debug. Where do I find the logs after applying the kernel parameter to my build and booting it?

tlaurion commented 3 years ago

Confirm the behavior after flashing with USB storage in.

tlaurion commented 3 years ago

confirmed after flashing again with usb storage in after oem factory reset

tlaurion commented 3 years ago

infirmed with HOTP security dongle while calling reboot from command line

tlaurion commented 3 years ago

confirmed with HOTP security dongle + usb storage while calling reboot from command line

tlaurion commented 3 years ago

infirmed while flashing rom and removing usb storage prior of rebooting. confirmed while rebooting having USB Security dongle in prior of reboot from command line confirmed that no keyboard with no USB devices connected in on calling reboot from command line.

So issue is not USB related.

tlaurion commented 3 years ago

Could it be linked to s3 resume suspend codepath being broken with new cross-musl-make compiler switch? (#608?)

alex-nitrokey commented 3 years ago

Mh, dunno, I never experienced such issue as in #608 and I am not sure what exactly causes it, so it is difficult for me to clearly see any connection, but could be :shrug:

tlaurion commented 3 years ago

Just saw at at... something something popping up prior of Heads banner about keyboard issue... Seems like to be coreboot related and linux not being able to initialize it...

@alex-nitrokey Have you tried putting the i8042.debug changes?

alex-nitrokey commented 3 years ago

Have you tried putting the i8042.debug changes?

Unfortunately, not. I hadn't as much time as I hoped to. I tried to fix/test some quick stuff today, but won't get to this keyboard issue before next week, I am sorry :(

alex-nitrokey commented 3 years ago

I played around with current master builds and did not experienced this issue anymore. Is it possible that it is gone by a change of the last weeks?

alex-nitrokey commented 3 years ago

Looking at the commits my best guess is the gnupg version

alex-nitrokey commented 3 years ago

Well I am not so sure anymore... I think they are more seldom with current master but I still get the keyboard freezing sometimes. It is strange...

alex-nitrokey commented 3 years ago

Have you tried putting the i8042.debug changes?

@tlaurion can you please go into detail how I would apply and use this command? I tried putting it in coreboot config CONFIG_LINUX_COMMAND_LINE, but I don't know if this is right nor how I could use it, especially once the keyboard freezes.

tlaurion commented 3 years ago

@alex-nitrokey confirmed you were lucky. Testing next PR merge here and confirms random keyboard unresponsiveness https://app.circleci.com/pipelines/github/tlaurion/heads/273/workflows/3ce7f868-50fb-420d-a45a-ec3048e350f3/jobs/296/artifacts

tlaurion commented 3 years ago

Have you tried putting the i8042.debug changes?

@tlaurion can you please go into detail how I would apply and use this command? I tried putting it in coreboot config CONFIG_LINUX_COMMAND_LINE, but I don't know if this is right nor how I could use it, especially once the keyboard freezes.

Would need to get the kernel to be really verbose on that line too... best debug output is from qemu board linux and coreboot configs, which is what is the most verbose we have. Trick would be to have mount-usb called and output everything there and mount read only to catch what is happening when unresponsive....

@alex-nitrokey Trick would be to call those as a debug hack prior of calling gui-init I guess since its platform specific. @MrChromebox you dont have this random issue on other platforms, right?

tlaurion commented 3 years ago

@alex-nitrokey : haaaaa, forgot I worked on producing a x230-debug config in the past

MrChromebox commented 3 years ago

@tlaurion no ps2k issues on librem devices of which I'm aware

alex-nitrokey commented 3 years ago

Just for the record: I had assumed here that the keyboard issue might got solved. It didn't. Additionally, I tested today a bit more with different coreboot/linux versions. None of them worked correctly regarding this issue.

I tested coreboot 4.8.1/4.12 and linux 4.14/4.19 repectively (resulting in four builds). So it seems that there is no connection in regard.

tlaurion commented 3 years ago

@alex-nitrokey : we are here in the troubleshooting path https://github.com/osresearch/heads/issues/765#issuecomment-652260072, which I haven't investigated yet, but confirm coreboot/heads giving error on screen just before GUI when keyboard is not recognized anymore. Logs would be helpful.

Suggested hack to facilitate log grabbing

tlaurion commented 2 years ago

@alex-nitrokey fixed under #1015

tlaurion commented 2 years ago

1015 merged