x230 performs unstable after flashing different Heads-Rom

[ghost]

Yesterday I internally flashed heads-hotp.rom on an existing heads-install to my x230.The flash went without error notifications and in order to reset the TPM I rebooted and after that the x230 started to emit a constant beep which only would stop after removing all power sources.Since then the x230 is performing unstable,like many browser crashes,sudden switches into screen-lock-mode.Checked different OS's,but no difference.Made an new external flash of the upper chip with x230-flash.rom,but it doesn't make any difference either.Since then on boot there is a notification above the Heads-logo that says "Invalid PCI ROM Header:Expecting 0xsomething...0xsomething".I am almost sure that this notification didn't appear before the flashing of the Heads-hotp.rom.The x220 running with Heads and Qubes OS is performing perfectly and there is no such notification.

[tlaurion]

Which commit? There were no change on neither coreboot nor Linux modules on master.

The PCI warning you have is in link with the absence of blobs for video and was always there.

Seems like RAM issue to me?

[ghost]

@tlaurion, I did an OEM-Factory-Reset on the x230 and it worsened the issue.Now the reset of the TMP is not working anymore,getting this output after trying it https://postimg.cc/T5dQs2VD.In between I reflashed stock Bios and then reflashed heads-x230-flash.Then I flashed the 12MB heads-rom internally shutdown,poweredup and then again I got the constant beep,removed the powersource and after rebooting I got this notification https://postimg.cc/cgV55GD7.Then after another powerup it booted into the x230menu,but still setting up TOTP didnt work.The setup for the x220 is also messed up now because the OEM-Factory-Reset on the x230 required that the GPG keys on the x220 to be renewed and somehow I cannot get the x220 running.Tried already the "add GPG keys from standalone Bios and reflash" option but it didn't work.Right now I use stock Bios on the x230 and will try to externally reflash the x220 heads-rom and see how that works.What's going on?On one day the machines work perfectly and then all seems messed up. As I already said,it started that I flashed the heads-x230-hotp***.rom version on the x230.Before that everuthing ran fine.What do you mean by something being possibly a RAM issue?

[ghost]

@tlaurion, I externally reflashed the x220 with the original heads-rom I still had.It boots into the menu,but there is a TOTP mismatch which I cannot fix.Already tried the suggested troubleshooting measures like "date -s HH:MM:SS" "-w hwclock".But not working.Although Qubes OS boots and works.Saying because since today the x230 doesn't decrypt the harddrive anymore.

[ghost]

@tlaurion ,the x220 is up and running.The TOTP is in sync again with the authenticator app on the smartphone.But I wonder whats up with the x230?

[tlaurion]

@userongihu You mix a lot of things here :)

1- On boot, heads verify that the TPM counter is valid (/boot counter), that that TPM ownership (TPM counter) matches with the counter on /boot. When upgrading firmware, user needs to reown the TPM. That would be the first screen shown when booting Heads after an external flash/internal flash with invalid counter. At that step, if you do not own TPM, you have to own it (even more if you reflashed OEM bios). To do so: Options, TPM, Own TPM or similar. That will regenerate a new Qr code, and requires that the time is in GMT under Heads, which can be resetted with instructions from the entry TOTP mismatch under menu. 2- You should not reown the laptop with OEM-Factory-Reset unless you want to wipe your USB Security dongle, reinject a new public key inside of the rom, retake measurements and seal those in the form of a Qr Code (and HOTP if enabled). For information, the TPM is resetted with new passphrase only on its last step, which is not and will cause the error you are encountering since the TPM counter is invalid, requiring of you to own it first. The goal of that OEM-Factory-Reset script is to regenerate keys on the USB Security dongle, which you should have backup of, and insert your backed up public key only if reflashing a clean ROM. Otherwise, updating from withing Heads would take care of exporting your keyring and reimporting it in upgrade. That will still invalidate TPM measurements and trigger a TOTP resealing, generating a new Qr Code which is GMT timezone based on laptop and time synced. (blog post, heads-wiki PR ) 3- TPM released Disk Unlock key depends on valid sealed measurements inside of TPM NVRAM space and valid /boot counter. This requires valid ownership of TPM (point 1), setting a new boot default as if you updated QubesOS dom0 components. To do so, go into Options, other boot options, show boot options. From there, select first grub entry (dynamic), say yes to save new boot default, y to save disk changes and add a disk unlock key injected by TPM. Your Disk Unlock Key is not released by TPM because measurements are invalid. This is normal since you haven't resealed TPM secrets (nor reowned the TPM after all those firmware changes). Disk Unlock key will be released by TPM only if TPM measurements + LUKS header + TPM NVRAM Disk Unlock key passphrase are all valid, otherwise printing PCRs values for you to investigate which parts of the firmware are different. In your case, PCR 2 (rom content since reflashed) and 7 (your public key) would be different, resulting in a mismatch. (example, PR for better doc)

[ghost]

@tlaurion Bit strange,booting and running Qubes on the x220 works,but when booting another distro on the x220 I do get this output https://postimg.cc/hhYmxz96.The TOTP sync between x220 and the authenticator app works no matter which distro boots. Thank you for your suggestions on how to set up a default boot config.Up to now,when booting Qubes,or any other distro, I just go through the options to get it to boot,because I am happy enough that it works.It happened too often,like in the case of the current problems with the x230,that changing something that already worked,messed it up due to my lack of knowledge.

[tlaurion]

@userongihu you have to own TPM to set a new TPM counter. Please do this and your problem will be gone. 1- Own TPM (Options, TPM) 2- Sign boot config (Options, GPG) 3- Set new boot default (Options, show boot default, select first option, then save to disk, Set Disk Unlock key to be released by TPM) 3.1 you will then: Provide Disk Recovery Key passphrase, renew/change Disk Unlock Key passphrase 3.1 You will then: Detach sign boot boot + /boot with USB Security dongle

After having own your computer once, Heads is pretty straightforward into guiding you when you do a firmware upgrade and dom0 QubesOS core updates. Booting a new default after upgrading dom0 will say that the default boot option changed, asking you to renew/change your boot default accordingly, redoing 3.1 and 3.2 in a guided way.

I'm sorry you had doubts, but we are drifting of the main question which was the cause of this issue. Scripts inside of Heads changed in time. Not used coreboot version nor linux version as of now, where only coreboot hardware init change would have impacts on your observed instabilities. Which leads me to reask: are your ram modules still good, well connected? Swap them around. You seem to have done on a lot of ROM related changes thinking that the rom became unstable, while testing with different OSes (kexec into linux distro passes control of hardware to the OS...) so if there is still instabilities, Occam razor would point in the direction of hardware issues. And hardware issues would speak more louder inside the OS, from dom0 through sudo journalctl -xe after a bug appearance, sudo dmesg etc, more then changing everything that low level without a clear hypothesis to verify and consequent tests to isolate a cause.

[tlaurion]

Do 1,2,3 above. That would resolve your current Heads issues.

Then here are my requirements to isolate behavior causes which are thought to be hardware based. Otherwise, we can only speculate:

Which commit? There were no change on neither coreboot nor Linux modules on master.

cat /etc/config under heads. Spot GIT_COMMIT. (While I had no other report of instabilities on desired upgrade coreboot and kernel for the x230). Should not be at cause and should be investigated after OS debug traces revoking theory of RAM/disk issues after a bug was encountered see below.

The PCI warning you have is in link with the absence of blobs for video and was always there.

Seems like RAM issue to me?

In OS: sudo journalctl -xe and sudo dmesg and then other troubleshooting paths can be thought.

[ghost]

@tlaurion Maybe I should add that Qubes is the distro that i use by default on the x220.In this case I changed the harddrive in order to boot Debian on the x220 for trying to flash the x230 again,this time using the original heads-x230-v-***.rom that I used when I flashed Heads for the first time to the x230.And lots of thanks to you for answering my questions in such a detailed and comprehensible way !Just noticed that you posted,while i am writing this

[ghost]

I will have a look at your sugestions and will report as soon as I have results.

[ghost]

@tlaurion Good news,the x230 works again.TOTP is in sync with the auth. app,Debian boots and works.As far as the browser crashes etc. are concerned,it seems that you were right,for I removed one Ram card and all the OS related glitches were gone.When I installed Heads for the first time I only used one Ram card and the glitches started around the time when I added another Ram card to the x230,they were of diffrent vendors.Maybe that was part of the problem.But it all works now.What I still need to do is setting up Debian as boot default. Will report.

[ghost]

@tlaurion ,one non-Heads related question:I upgraded to the latest kernel.Now after booting there's a notification that says that there is almost no space left in /boot.Will it affect the functionality of Heads if I remove the 4.19 kernel,which isn't used anymore.Would the usual command for kernel removal be sufficient?

[tlaurion]

This is related to maintaining debian systems (QubesOS creates a 1gb+ /boot partition so also unrelated to QubesOS), unrelated to this particular issue nor Heads, as you guessed. Closing issue as fixed and linked to physical memory issue.