keyboard backspace bug for x230-hotp-maximized (x230t)

[Kawaiisu]

So I was excited to get heads installed and my nitrokey setup. But then realised I have some sort of bug on all three laptops. No matter the OS, if its linux it locks the screen or just doesn't do anything. Same issue on 1 x230 tablet and 2 x230 notebooks.

How can I go about fixing this? I'm not a seasoned developer so please have mercy on me.

Thank you

[tlaurion]

@Kawaiisu what is your backspace bug?

I had the following bug on x230t but in OS (QubesOS) not under Heads, if your bug is that the backspace is not linked to the right keymap for some reason: https://github.com/QubesOS/qubes-issues/issues/3306

Please describe actual behavior and expected behavior, which OS, and if behavior is under Heads or not. The operating system being kexec'ed, Heads should not have any influence on the OS behavior. We are interested in the behaviors happening inside of Heads.

[Kawaiisu]

Hi,

I went to powershell from Heads and they backspace and backspace keys seem to be fine.

Then I tested debian 10 live and found that the backslash and pipe key "\ |" seems not to function. As well as Backspace.

Again in Qubes OS, same as debian 10, the backspace or backslash doesn't seem to function.

On Tails backslash does nothing but backspace in this instance locks the screen.

I visited this: https://github.com/QubesOS/qubes-issues/issues/3306

Am I supposed to overwrite xmodmap_x230t.txt from Dom 0?

But what about every other operating system? Doesn't it mean there's something I need to do in heads to make sure its permanent across all operating systems? We use many live operating systems.

Thank you.

[tlaurion]

@Kawaiisu The problem was never resolved. Last time I checked with QubesOS, fix was temporary, and was not able to map it correctly over reboot and was randomly showing back. Looked for help, found none, gave up.

AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all). The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.

If you find the solution elsewhere, please link to the fix. It seems that no one cared, including myself, since I do not use my x230t but to test Heads builds. AFAIK that QubesOS issue included all the pertinent information that I had found from that time. If you find something new, please ressucitate that issue, since as you discovered, is still present. Also, as you saw, the issue is not having any relevance to Heads, since the behavior is not present.

So the issue here is irrelevant.

[fhvyhjriur]

AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all). The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.

If you find the solution elsewhere, please link to the fix.

@tlaurion Could you use the x230t coreboot config for x230t? Skulls is building separate images for x230 and x230t because the devices are not exactly same. https://github.com/merge/skulls x230t support was merged in coreboot on 03.2020 here: https://review.coreboot.org/c/coreboot/+/38482

It would also make more sense for people who switch from skulls to heads to flash a x230t image when they had before also a x230t skulls-coreboot image.

[tlaurion]

@ fhvyhjriur Heads is a community project. There is a porting guide on heads-wiki. As you saw for #925 its not really useful neither for myself nor others to do ports for hardware I do not use. The people that have needs should be the one pushing changes. This is mentality of open source and is called Selfish Contributionism. Maintainership is the main problem of open source projects. It would be counter productive that I accept to bring a board to a project I do not use, for whichI will then be asked to maintain. It won't happen.

Consequently, I will not do the port for x230t but will invite x230t, t400 and other not supported board owners to propose pull requests.

And will hold my stance on maintaining and trying to continue to do quality work on my community time, for which i'm not paid for. Hope the message reached its audience.

Free software doesn't mean free as in free beer. Everyone has 24h in a day. If it's important for you, you will take the required time to make it happen, asking questions, proposing code; not requiring others to do the work at your place. This is how it works.

I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.

Letting people take their fish lines instead of providing unlimited fish supply: cp config/coreboot-x230.config build/coreboot-ver/.config cd build/coreboot-ver/ make menuconfig (do config changes needed) make savedefconfig cp .config ../../config/coreboot-x230t.config cd - cp -r boards/x230/ boards/x230t/ mv boards/x230t/x230.config boards/x230t/x230t.config vi boards/x230t/x230t.config (change path for coreboot-x230.config to coreboot-x230t.config) make BOARD=x230t (adjust until it compiles) git checkout -b x230t git add board/ config/ git commit git push create PR.

[fhvyhjriur]

@tlaurion I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.

You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.

EDIT: When i wrote the text, your last edit of your text did not had this line: 'I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.' Yes, this was the reason why i wrote what i had wrote before.

[tlaurion]

@tlaurion I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.

You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.

I did'nt know. That would then be linked to x230-maximized images, being pushed to coreboot newer version, which are still not merged upstream under Heads. My comment is still relevant. Code should be borrowed, modified, tested functional, and then proposed for merged, tested and then upstreamed by the people who need the changes.

[fhvyhjriur]

@tlaurion I watched https://fosdem.org/2020/schedule/event/selfish_contributor/ My "selfish motivation" bringing up a idea how to maybe help @Kawaiisu and all other heads users is because heads is a way to do important work as a investigative journalist that travel around the world and have to leave the device for a moment at some place that is not secure. @Kawaiisu is a new user that have register fresh to github to report this problem. So i expect that this user needs heads to do something important to the society and needs help with that. I want to live a life in a knowledgeable society with free access to free(copyleft) knowledge. I am also "selfish motivated" to support old hardware because of freedom(low purchase price, low to none dependencies to startup based on closed source software) and environmental benefits. My "selfish motivation" in for example getting broken printers from electrical waste and soldering out their 8MiB and 16MiB spi chips and putting those SPI chips in thinkpads that have 4MiB or less. To get heads support for such devices is something i am interested in after learning facts about this world. I am not a developer and i am not good at writing code. I depend, like mentioned in the https://fosdem.org/2020/schedule/event/selfish_contributor/ video, mainly on commit messages that explain what the commit have done. And because T400 was mentioned - i have spend happily some hours into taking a T400 apart and writing with a external programmer some heads testing-image and reporting if its working or not. I would do this work again to help adding support for T400 and other devices that are not supported in heads recently. Adding heads-support for all x8664 devices that coreboot support and that can run in perfect case with 100% free software except cpu-microcode is part of my "selfish motivation". It would be my "selfish motivation" until RISC-V based portable devices are available at the same price and amount like ~10 year old Thinkpads that heads support recently. At least until devices like the https://en.wikipedia.org/wiki/Novena(computing_platform) and now https://mntre.com/reform get the same produced amount like the thinkpad 00, 10, 20 and 30 series combined together.

At some parts of this world the 11.01.2021 have begun. This is the 8th year since the death of https://en.wikipedia.org/wiki/Aaron_Swartz . My "selfish motivation" is to help the world to be a better place for people with such a free mind like Aaron had.

[Thrilleratplay]

@fhvyhjriur

Could you use the x230t coreboot config for x230t?

Yes. My Heads test device is a 230T. As far as I can tell is the only difference between the x230 and x230T coreboot configurations is gpio33 changed from output to input. This probably has to do with using the touchscreen as an input during boot up.

[tlaurion]

@Thrilleratplay did you figured out why backspace was not functionning properly on Qubesos and other OSes?

[tlaurion]

@Kawaiisu first question first. Is the bad behavior present with stock Lenovo BIOS on live cds?

@fhvyhjriur lookin at coreboot changes, I doubt sincerely that the changes would resolve the keyboard issue.

With that first question here answered, avenues of solutions can then be nrrowed down.

Vivid memories for Swartz. Sympathetic to the cause. Thanks for reminding me why we do this. Just remember time is a limited resource, I don't intend to be rude.

[Thrilleratplay]

@tlaurion I haven't had an issue with the backspace. I'll try the latest build tomorrow

[Thrilleratplay]

@tlaurion Open source projects are rife with burnout and you put in a considerable amount of work to Heads. It is important to be reminded why you it. The personal reasons that drive you to contribute so much to Heads are your own. I understand the importance and do what I can with my time, resources and energy but pales in comparison to you do to keep this project going. Thank you.

[Thrilleratplay]

@Kawaiisu I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries.

What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards.

I still have to test this myself.

[Thrilleratplay]

@tlaurion @Kawaiisu I may have reproduced this. I am confused on whether it is just the backspace key or all keys that do not work? I flashed the latest x230 maximized build artifact from CircleCI and everything seems to work fine but occasionally after a reboot, the keyboard does not seem to be initialized. It also only seems to happen when rebooting from the emergency shell either with the command reboot or ctrl-alt-del.

EDIT: Oops, I used a test build. Let me try one from master.....and osresearch does not allow/have any projects in circleCI because there is only a 404 when I click on projects.

EDIT 2: Updated master on my fork and used the CircleCI build heads-x230-maximized-v0.2.0-1006-g6bc40d7.rom. If I boot into a USB image of Debian 9, I can go into a prompt and still use the backspace and pipe characters on the keyboard.

[ghost]

@tlaurion @Thrilleratplay

i remember this bug with backspace key sorry but its not only Qubes same issue with x220/x220t (flashed year ago) it works in heads shell it lock screen on gnome desktop (lastest ubuntu) it doesn't work everywhere in terminal/chat/text editor never seen this bug on clean coreboot or stock bios. replaced keyboard - nothing changed maybe really xev or xmonad can help I use classical six row kbd

I would never give more than $ 50 for these laptops, ebay disappointed me a lot. out of five, I have only one alive (librebooted x200 from Vikings)

[Kawaiisu]

RIP Aaron :(

Thank you all so much for your generosity.

Sorry for my delay. I hope you're all finding ways to prepare and maintain your safety during these trying times.

@tlaurion : you asked "Is the bad behavior present with stock Lenovo BIOS on live cds?"

I will double check this on a stock device and post my results today.

Also I think you're correct that there isn't this problem on the notebook non-tablet version of the x230. I'll double check this as well.

@Thrilleratplay you said "I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries."

No mods. I did do a backlit keyboard on one, but went back to stock keyboard.

@Thrilleratplay: "What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards."

Laptops were purchased from a recycler. But there was spanish post-it notes from the previous owners. Its possible they came from South America, would this matter? What could I do to figure this out? I think the recycler would not tell me for competitive reasons.

I will go through in detail everything else mentioned above in case I missed anything.

[Thrilleratplay]

@Kawaiisu I ask about where they are from because there are regions variants of keyboards (Japanese, Russian etc). Normally these are obvious, other times it could be subtle like the currency character not being $ if you are in the US. A few keys could be mapped differently on these variants for backspace and pipe. Although, I think the backspace would be the same on all of these.

[Kawaiisu]

@Thrilleratplay @tlaurion

Okay I've tried both tails and debian 10 live on both stock x230 notebook non-tablet version and x230 tablet versions. Backspace and and backslash work normally. I also swapped out the tablet with a backlit keyboard and backspace and backslash still work as expected.

Also not sure if it will become relevant but want to note it so I don't forget: Another X230 notebook I flashed using make BOARD=x230 and x230-flash rom creation process. On that system backspace and backslash work just fine.

The system that's giving me the problem is an x230 tablet and I used make BOARD=x230-hotp-mazimized. Not sure if the rom difference is relevant.

I am going to now flash the stock x230 tablet and notepad with the x230-hotp-mazimized rom and will report if the backspace-backslash problem comes back.

@tlaurion about whether I have country specific keyboards, they seem to be all US, except 1 backlit keyboard on the problem computer does have a random extra euro symbol on the 5 and % key. So three symbols on this one key. Which is strange. The other backlits don't have this and seem to be US-only. Could be something here.

Okay I'll get to flashing and report back my findings.

But first I have a question about preparing the x230-hotp-maximized roms.

In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:

"Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop."

I know I'm doing something really noob here. A hyphen or space or something is missing or needs to deleted, please have mercy on me ^_^.

To produce the roms while getting around the error we skipped this part and manually ran wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin then made the roms after.

But if I know why the error was happening this will speed up our process.

Thank you. I'll get to flashing now.

[Kawaiisu]

@0rb677 @tlaurion @Thrilleratplay @fhvyhjriur

Greetings everyone. I'm reporting back my findings.

I newly flashed two laptops, a stock x230 Tablet and a stock x230 Notebook non-tablet.

The notebook's backspace and backslash work fine. However on the Tablet backspace and backslash don't work as intended. Locking the screen in Tails for instance.

I did test both stock laptops prior, they were both working as intended on multiple OS before the flash.

So at this point I'm stumped, is this a heads problem or something else?

Not really sure how to proceed here. Any advice would be greatly appreciated.

Thank you all for your time and insight. Be safe.

[Thrilleratplay]

@Kawaiisu Just to clarify, all of your x230 work as expected but the x230T still has an issue with the backspace key? What OSes have you tried after flashing Heads?

Lets try to figure out what the OS sees (link to helpful answer). At the moment I am on a x230 that is running coreboot and Arch Linux.

If I execute sudo showkey and press backspace. The key code is:

keycode  14 press
keycode  14 release

sudo showkey -s and press backspace. The scancodes is:

0x0e 
0x8e 

If the same OS and version of Heads is installed on both your x230 and x230T, it would be more beneficial if you compare the the results between those machines. Hopefully this will narrow down the issue.

[Kawaiisu]

Hi @Thrilleratplay

Yes you're correct on the x230 backspace and backslash works fine with Qubes.

But x230T with Qubes backspace and backslash doesn't work.

On x230, running sudo showkey and pressing backspace shows:

keycode 14 press keycode 14 release

Then running sudo showkey -s and pressing backspace shows:

0x0e 0x8

Then running sudo showkey and pressing backslash shows:

keycode 43 press keycode 43 release

Then running sudo showkey -s and pressing backslash shows:

0x2b 0xab

On x230T, running sudo showkey and pressing backspace shows:

keycode 152 press keycode 152 release

Then running sudo showkey -s and pressing backspace shows:

0xe0 0x12 0xe0 0x92

Then running sudo showkey and pressing backslash shows:

keycode 139 press keycode 139 release

Then running sudo showkey -s and pressing backslash shows:

0xe0 0x1e 0xe0 0x9e

Does that help?

[Thrilleratplay]

@Kawaiisu It sounds like a bug in Qubes and now that you know what the keys are, remapping seems like the easiest fix. I am not familiar enough with it to suggest an appropriate work around. @tlaurion Can you suggest a way to globally remap keys in Qubes?

[Kawaiisu]

@Thrilleratplay @tlaurion Want to give update. The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.

Also on the 230T, in heads powershell backspace works correctly. So I'm confused, why do all OSs work on a stock X230T, but when I flash heads, suddenly no OSs work?

Are we sure this is a Qubes specific problem? Isn't Qubes natively Xen? Should I test more OSs?

[tlaurion]

@Kawaiisu i'm confused on your conclusions. You say that QubesOS, Tails and Debian 10 live OSes: the behavior is the same. I guess the behavior is bad? Or you mean by flashing Heads, the behavior changes to bad?

@Kawaiisu : can you send me a rom backup of lenovo x230t? I cannot guarantee when I will invest time in this, but my first debfugging steps, as said prior, would be to test behavior on stock BIOS. My hypothesis is that OSes are detecting keyboard and set mappings based on detected x230. Which doesn't match x230t. Another note: QubesOS is based on Xen, yes, but the underlying dom0 OS for QubesOS 4.0 is fedora-25. So issues disovered here are fedora-25 related for QubesOS. Or related to Xorg keyboard, or underlying mapping to keyboard config.

@Thrilleratplay I left the ticket as unfixed under QubesOS with all steps I took in the past trying to fix the issue. As of now, i'm still into uncertainty on what is the problem to replicate. On my past experience on X230t with QubesOS, the behavior was random. It was functional at first install (I could change LUKS key passphrase doing backstpace) and then after OS install, I couldn't. The issue I opened (QubesOS 3.2) was closed because noone else was confirming the issue. So the user base of X230t seems quite small. Noone else replicated the issue; noone +1. I investigated as far as I could. And left the x230t alone.

The heads related ticket is : https://github.com/osresearch/heads/issues/827#issue-692472819 The QubesOS related issue is: https://github.com/QubesOS/qubes-issues/issues/3306

I think the problem is generalizing to all OSes. And the fix to test is this: https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-359933561 Unfortunately, it seems that it was not persistent(the problem came back on next boot? I can't recall. 2018, after all!), since I after that posted https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-366265534 ....And then, lost interest in the X230t, which I use to test flashes, and where Heads backspace works, where there is no added layers of overcomplications or guessing and overides of keymaps.

So I will ask again:

1. Is the behavior present on stock Lenovo bios for X230t?
2. Is the behavior limited to Heads?

If the behavior is limited to Heads, which is a duplicate of the X230 coreboot config, then maybe coreboot exposing the board as a x230 in dmi tables might be the culprit. Maybe the remapping of keys in OSes are basing themselves on exposed X230 mappings instead of X230t.

Please give feedback on 1 or 2 above. Instead of trying to resolve the issue and waste time. Troubleshooting steps from simple to complex is key, else you will loose yourselves. If we know the behavior is different between installed OSes from stock BIOS vs Heads, then we might be able to get the differences in assumptions and blame the x230 config inside of the x230t board. And start from there.

If I missed your conclusions, please re clarify and sorry to have missed them; I have not see clear statement of differentiation between 1 or 2 being the cultprit.

[Thrilleratplay]

@Kawaiisu I asked what OSes you had tried, as the respond was only pertained to Qubes, I assumed the issue was isolated to Qubes.

I booted Debian 9 on my x230T with maximized Heads and do not have this issue. The shell I entered from the Debian Graphical install does not have showkey. The best thing I can find is to run more /proc/bus/input/devices. On my x230T, the keyboard name is "AT Raw Set 2 keyboard". Interestingly, my x230 is "AT Translated Set 2 keyboard". Looking at them, they look like they are interchangeable.

Steps to take:

• Boot into Debian using the USB
• Go to a shell
• run more /proc/bus/input/devices, scroll until you find your keyboard.
• Google search the name and maybe the version ID to see if there are any issues with that specific keyboard.
• Afterwords, for shits and giggles, swap the keyboards on a working x230 and the x230T and see if that fixes it. If this works and no one knows what to correctly map the bad keyboard, it may be easier to buy a replacement keyboard at this time.

@tlaurion I saw the solution you posted in Qubes issues #3306 using xmodmap but, unless I am mistaken, this would only work in X11, not in a terminal if x11 failed to start or the user enters another TTY session by pressing Ctrl-alt-F2 (this may not be an issue in Qubes).

[Kawaiisu]

@tlaurion apologies, let me correct my poor grammar:

The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.

You asked:

1. Is the behavior present on stock Lenovo bios for X230t?

The bad behaviour isn't present on a stock lenovo bios for the x230t. I test qubes, debian live and tails.

2. Is the behavior limited to Heads?

Yes the bad behaviour of incorrectly mapped backspace and backslash only appears only after flashing.

I tried to upload .zip directly to github, either its not supported or my browser doesn't allow. I was able to upload here:

https://ufile.io/3vrd09o2

Inside is both heads that I flashed to the x230t and the factory bios before should you need.

[tlaurion]

So works on stock bios but not heads. Now we talk.

[fhvyhjriur]

@Kawaiisu Could you give the skulls x230t image a try and report if you have the same issues? https://github.com/merge/skulls/releases/tag/x230t-0.0.2 Probably the best way to do the test would be to first install the x230t oem backup you have made at some point, test if the issue is gone with the OEM again (never know if EC does the problems) and then follow the skulls installation howto. Would be interesting to know if the issue is gone with that skulls-x230t-image that is made for x230t. If the issue is gone with the x230t skulls image, you can flash the skulls-x230 https://github.com/merge/skulls/releases/tag/x230-0.1.11 release to the x230t to find out if it have to do something with that and the issue appear again.

[Kawaiisu]

@Thrilleratplay

Upon your suggestions I ran on debian 10:

more /proc/bus/input/devices

It showed on the x230t:

AT RAW set 2 keyboard" and ID: "ab84

I googled around and nothing came up relevant to this issue.

I swapped out the keyboard from a x230 notebook where the backspace is working on to the x230t, and the backspace still doesn't work. I ran more /proc/bus/input/devices and the readout is identical to before AT RAW set 2 keyboard" and ID: "ab84

You said heads maximised is working correctly with your x230t, do you think it's possible I built heads incorrectly?

Beforehand when I built heads, I was using make BOARD=x230 and x230-flash, I had to copy the original OEM roms to the flashing device and unlock them or remake them into new roms.

But there was a key difference when doing BOARD=x230-hotp-maximized, where I never unlocked the OEM roms from the original x230t. Instead the roms were downloaded. Is it possible I downloaded the wrong roms?

In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:

Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop.

I couldn't figure out what the custom invoke or call command was here so I tried to get around this step by manually running

wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin

then made the roms after.

Is it possible I missed a key step involving a difference in the way to make the roms for the x230 versus the x230t?

[Thrilleratplay]

@Kawaiisu The id is the same as the keyboard in my x230T, so that isn't it.

I am very confused. Step by step, list how you built the rom and flashed this device. I do not know how you were able to compile the x230-hopt-maximized without running that script or placing in your own extracted ME and GBE files. Line 65 is not meant to be uncommented, it is an instruction, you are supposed to run that command before. Also, didn't you flash the same ROM to both the x230 and x230T? At this point in the ticket I am lost to what device has what version of what ROM and what changes have been made, if any, to the Heads build.

Also, I realized I have only been flashing the x230-maximized, I do not own a Nitrokey/Librem key so I cannot test the HOTP functionality. Try flashing the FULL ROM I linked a few days ago, the CircleCI build heads-x230-maximized-v0.2.0-1006-g6bc40d7.rom. See if that gives you the same issues. By full ROM I mean, do not use the Heads flash.sh script, use flashrom -p internal -w <filename> or split the file into 8mb and 4mb chunks and flash externally.

[Kawaiisu]

@Thrilleratplay

Heres my butchered step by step. I was referencing maybe 6 different guides to try and make this work:

Build/Make Heads On Audited System:

Download all repositories and clone heads

git clone https://github.com/osresearch/heads

sudo apt update

sudo apt install -y build-essential zlib1g-dev uuid-dev libdigest-sha-perl libelf-dev bc bzip2 bison flex git gnupg iasl m4 nasm patch python python2 python3 wget gnat cpio ccache pkg-config cmake libusb-1.0-0-dev autoconf texinfo ncurses-dev doxygen graphviz udev libudev1 libudev-dev automake libtool rsync innoextract

Clone ME_Cleaner

git clone https://github.com/corna/me_cleaner.git

At heads directory run command:

make

at heads/blobs/xx30 run command:

wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin

I thought this step was recommended in the readme file in the blobs/xx30 directory at line 27, but seems I did not understand. -Kawaiisu

at heads top directory run command:

make BOARD=x230-hotp-maximized

The result: 3 new rom files named heads-x230-hotp-maximized-v-dirty-bottom, -top and -nothing, at 12, 8 and 4MB.

On Flashing System:

If Using Debian Live USB:

open terminal to install flashrom utilities

sudo apt install -y \ bison \ build-essential \ curl \ flashrom \ flex \ git \ gnat \ libncurses5-dev \ m4 \ rename \ zlib1g-dev

Verify Flash Device Communication:

First physically read and note if possible both the ID of the top and bottom chips starting with "MX"

Use following command three times to verify read of top chip:

sudo flashrom -p ch341a_spi

it finds multiple versions of the chip. If you are able to read the print on your chips, there are numbers that match the output of the above command.

Once read verifies you have a good connection back up the top chip:

sudo flashrom -p ch341a_spi -r factory_top-1.bin -c "MX25L3206E/MX25L3208E"

sudo flashrom -p ch341a_spi -r factory_top-2.bin -c "MX25L3206E/MX25L3208E"

sudo flashrom -p ch341a_spi -r factory_top-3.bin -c "MX25L3206E/MX25L3208E"

Verify perfect copies as the chip reader connection can be flaky:

sha512sum factory_top-*

Now flash the top ROM.

time sudo flashrom --chip "MX25L3206E/MX25L3208E" --programmer ch341a_spi --write heads-x230-hotp-maximized-top.rom

Once you see "verified" connect SPI reader to bottom chip.

Use following command three times to verify read of bottom chip:

sudo flashrom -p ch341a_spi

Once good connection confirmed backup chip.

sudo flashrom -p ch341a_spi -r factory_bottom-1.bin -c "MX25L6406E/MX25L6408E"

sudo flashrom -p ch341a_spi -r factory_bottom-2.bin -c "MX25L6406E/MX25L6408E"

sudo flashrom -p ch341a_spi -r factory_bottom-3.bin -c "MX25L6406E/MX25L6408E"

Verify Hashes:

sha512sum factory_bottom-*

Once verified flash the bottom chip:

time sudo flashrom --chip "MX25L6406E/MX25L6408E" --programmer ch341a_spi --write heads-x230-hotp-maximized-bottom.rom

Now install RAM and boot laptop to setup LibremKey. Heads should be installed with IME killed.

Guide End

I will try your .rom file. Sorry I missed this beforehand.

[Kawaiisu]

@Thrilleratplay I've used your .rom file via mounted usb and running internal flash from heads. Unfortunately, the backspace problem still exists in tails and qubes os.

[Kawaiisu]

At this point in the ticket I am lost to what device has what version of what ROM and what changes have been made, if any, to the Heads build. @Thrilleratplay

I've been using heads-x230-hotp-maximized-v0.2.0-993-g817e6a3-dirty.rom to flash both x230 and x230t. Is this incorrect?

And now I've used your x230-maximized-v0.2.0-1006-g6bc40d7.rom to flash the x230t which produced no change on the backspace problem.

[Thrilleratplay]

@Kawaiisu Thank you for taking the time to detailing what you have done. This ticket has discussed multiple devices, multiple OSes and multiple versions of Heads. Between all of the post, I could not keep track what has been tried.

Is this incorrect?

No. You have done everything perfectly correct as far as I can tell (again, thank you for the detailed write up). The only thing to mention is that download_clean_me.sh automates the steps to create a minified ME; this should be made clearer in the README, sorry. With Heads, do be "incorrect" you either cannot compile a ROM or if it does compile, the device will not boot. This issue is far more subtle.

At this point, I am stumped. It the backspace works correctly on Tails and Qubes with the stock BIOS but not Heads. The only thing I can suggest trying is flashing Skulls onto the x230 Tablet. If that works, then the issue is probably with coreboot that was corrected between version 4.8, used by Heads, and 4.13, current and used by Skulls. Our keyboards are the same for out x230 Tablet devices, you have tried the exact same ROM I use without an issue, the only thing left I can think of is a weird bug in coreboot that somehow has an issue with different versions of the EC.

Beyond that, I am out of ideas.

[tlaurion]

Same unresolved issue for x220t https://github.com/osresearch/heads/issues/698#issue-587284331

[Kawaiisu]

Hello @Thrilleratplay @fhvyhjriur

Per your recommendations, I've retested the backspace issue first on OEM and then Skulls onto the x230T.

On factory bios, backspace comes back, working in Qubes OS and Tails.

Then flashing https://github.com/merge/skulls/releases/tag/x230t-0.0.2, backspace and backslash work fine.

Then flashing https://github.com/merge/skulls/releases/tag/x230-0.1.11, backspace and backslash still working as expected on Qubes and Tails.

Hope this helps. I am available for anymore tests.

Thank you.

[Thrilleratplay]

@Kawaiisu Both the x230t and x230 builds of Skulls work? If so, I think a few conclusions can be made.

I doubled the commits related to x230t in coreboot, while there is one change that seems to be uniquely related the tablet version of the x230, names in the make file are simply aliases; meaning, within coreboot, there is no difference between the x230 and x230T other than name.

Heads currently is still using coreboot 4.8, Skulls is using a commit between coreboot 4.12 and 4.13, Between that time there were a number of (fixes for keyboards in libpayload)[https://review.coreboot.org/q/status:merged+libpayload+keyboard]. One of these may have fixed the issue for the x230T. You try to build Heads with a newer version of coreboot by manually changing the board config to version 4.11 or 4.13. In the related x220T issue, #698, @i-c-o-n suggested adding the kernel parameter i8042.direct=1, which may be easier to try first.

[i-c-o-n]

AIUI, there never was any issue with Skulls that any commit could have fixed, also Skulls uses SeaBIOS and nothing libpayload based. The difference with Skulls is that the keyboard had already been initialized once before Linux comes into play. With a Linux payload, the keyboard controller is left untouched until Linux tries to initialize it. There is also a Kconfig option in coreboot, DRIVERS_PS2_KEYBOARD. It does some random initialization that sometimes happens to make things work because it leaves the keyboard controller in a different state. Also slows the boot down, though.

[tlaurion]

https://github.com/tlaurion/heads/commit/79242a7abce6627d5541039bb00d918479ddba54 which artifact for x230-hotp-maximized will be available https://circleci.com/gh/tlaurion/heads/857?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Testing this over x230t would test https://github.com/osresearch/heads/issues/958#issuecomment-766230173 @i-c-o-n idea that DRIVERS_PS2_KEYBOARD inside of coreboot being specified might resolve the current issue.

[tlaurion]

@Thrilleratplay @i-c-o-n @0rb677 @fhvyhjriur https://857-103208611-gh.circle-artifacts.com/0/build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1033-g79242a7.rom including DRIVERS_PS2_KEYBOARD in coreboot as suggested by @i-c-o-n permitted to boot Tails from USB with all problematic keys working on test x230t machine.

Please replicate

Next test: testing QubesOS OEM image on the machine with that ROM.

[tlaurion]

Testing OEM disk image with a x230 keyboard swapped on x230t.

First boot of QubesOS:

• All keys work.
• shutdown.
• Hard reboot

Second boot of QubesOS:

• backspace doesn't work.
• showkey shows 152 for backspace as in https://github.com/osresearch/heads/issues/958#issuecomment-761826666 from @Kawaiisu testing trace.

Looking into backlog of x220t and other threads to see what was the other troubleshooting points and differences from coreboot/linux detection and how to fixate the proper de keyboard, for which correct initialization is random for unknown reason on x230t and x220t from coreboot+linux payload (heads), which is different, like said previously by others, then skulls or UEFI firmwares which really on a bootloader, not linux (to kexec into another ELF built binary).

Edit: @i-c-o-n suggested https://github.com/osresearch/heads/issues/698#issuecomment-765313861 adding i8042.direct=1 to kexec call (KERNEL_ADD under heads). Wil lcommit and push referring to this issue. Build on CircleCI for x230-hotp-maximized will contain the change. Feel free to replicate the commit locally and make this issue go forward by reporting your results if you are faster then I am testing and if you own, and currently want to use, a x230t.

[tlaurion]

Next step, if test build with previous fix doesn't do it is https://github.com/osresearch/heads/issues/698#issuecomment-808505315

[ghost]

@tlaurion, what is relevant now for x220? rebuilt the laptop from scratch and switched back from x220t to x220 trying to flash lastest dev build from circleci - heads-x220-maximized-v0.2.0-1035-gc87666e.rom and see how it works also will try to boot 4.1 from here with heads - https://qubes.notset.fr/iso/Qubes-20210327-x86_64.iso

p.s. screen artefacts again - no success. backspace works on x220 with 4.0.4 and heads maximized (not tablet)

[tlaurion]

Hey @0rb677 : I see you tried https://884-103208611-gh.circle-artifacts.com/0/build/x220-maximized/heads-x220-maximized-v0.2.0-1035-gc87666e.rom without success.... but not on good platform? Was that on an x220t? Read your post and it seems it was completely unrelated?

When a xx20t or x230t test result will be posted (working/not working ON THOSE PLATFORMS ONLY) the next step, if non working, is to backport https://github.com/osresearch/heads/issues/698#issuecomment-808505315 which I haven't had time to do. Please feel free to poke community if unable to do it yourself for community members if you do not own the platform in question.

The test to perform here is with the x220t (tablet): @0rb677: seems like you tested something that was not relevant to you on a non-tablet platform?

p.s. screen artefacts again - no success.

Was that supposed to be fixed here? I was not aware.

We are talking here about keyboard issues for tablets versions of x220 and x230 variants: your comments to this ticket are totally irrelevant/ Please comment other issues that are applicable?! This is neither x220 relevant nor related to a x220 issue: this is relative to x230 backspace (keyboard related) issue.

backspace works on x220 with 4.0.4 and heads maximized (not tablet)

How is this relevant to present issue? There is no x220 keyboard issue that i'm aware of? The relevant issue is for x220t referred above.

Please don't be a troll and read contextual information and reply in relative issues.

[tlaurion]

Please tag me to reopen. Since coreboot now deals with keyboard init through ps2, tests shows that x230 tablet is now not having this issue anymore.

[t-anon]

@tlaurion Tagging as requested. New HEADS user here - I'm still having this issue on my x230t with a fresh build of x230-maximized, despite CONFIG_DRIVERS_PS2_KEYBOARD=y in https://github.com/osresearch/heads/blob/master/config/coreboot-x230-maximized.config#L15. Please let me know how I can help get to the bottom of this.

Same exact issue as noted above - showkey shows 152 when I press backspace.

Interestingly, backspace works at the Qubes disk decryption screen, but fails after that screen passes.

Edit: Do I need to manually include the changes from https://github.com/tlaurion/heads/commit/d391e5d68aa2ed76652591e7f95ce32f87261c11?

Edit 2: Including the changes for x230-maximized didn't fix anything.

[tlaurion]

@t-anon : could you test booting Tails from USB? (Another OS)

The Qubes installation was made prior or after flashing Heads?

If the Qubes Disk passphrase promy here is having normal behavior, hypothesis is that generated X config is probably fixating something there.

This issue required a lot of troubleshooting in the past and conclusions were that the behavior was random. Working on boot then not on reboot.

I reread the whole issue and was pretty sure 4.13 coreboot was including proper initialization of PS2 keyboard.

So it goes back into retesting and giving as much info as possible.

Is there other x230t users that you could find in the issues that could be tagged here?

[t-anon]

@tlaurion on Tails backspace locks the screen. In general, it looks like I am exhibiting the same issues as @Kawaiisu was earlier (https://github.com/osresearch/heads/issues/958#issuecomment-757460080).

Unfortunately I don't know any other X230T owners.

Edit: Qubes install was after flashing Heads.

[tlaurion]

@t-anon: reviewing coreboot logs from Heads on Keyboard init (x230-maximized tested here on x230).

cbmem -c | grep -i keyboard gives on my side a serie of timeouts:

Keyboard controller output buffer result timeout
Keyboard controller output buffer result timeout
Keyboard controller output buffer result timeout
Keyboard controller output buffer result timeout
Keyboard controller output buffer result timeout
Keyboard reset failed ACK: 0xaa

On Heads linux side, doing dmesg|grep i8042 : input: AT Raw Set 2 Keyboard as /devices/platform/i8042/serio0/input/input4

What are yours under x230t prior of booting OS? Seems like coreboot ps2 keyboard workaround might not be doing exactly what is expected here. Might be a trail to follow. coreboot topic on ps2keyboard and code changes: https://review.coreboot.org/q/topic:ps2keyboard