Reboot from QubesOS fails if S3 standby was used since last boot after NK Heads upgrade 2.1 -> 2.4

UndeadDevel commented 9 months ago

A. Provide Hardware Details

1. What board are you using (see list of boards here)? NV41 2. Does your computer have a dGPU or is it iGPU-only?

[ ] dGPU
[x] iGPU-only

3. Who installed Heads on this computer?

[ ] Insurgo
[x] Nitrokey (originally)
[ ] Purism
[ ] Other provider
[x] Self-installed (internal upgrade to NK Heads 2.4)

4. What PGP key is being used?

[ ] Librem Key
[ ] Nitrokey Pro 2
[ ] Nitrokey Storage
[ ] Yubikey
[x] Other (NitroKey 3A Mini)

5. Are you using the PGP key to provide HOTP verification?

[x] Yes
[ ] No
[ ] I don't know

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

[ ] First-time flash
[x] Updating heads (update itself was successful, but a few regressions occurred, including this one)

2. If the problem is related to an update, how did you attempt to apply the update?

[x] Using the Heads GUI
[ ] Flashrom via the Recovery Shell
[ ] External flashing

3. How was Heads initially flashed

[x] External flashing (by NitroKey)
[ ] Internal-only / 1vyrain
[ ] Don't know

4. Was the board flashed with a maximized or non-maximized/legacy rom?

[x] Maximized (presumably)
[ ] Non-maximized / legacy
[ ] I don't know

5. If Heads was externally flashed, was IFD unlocked?

[x] Yes (presumably)
[ ] No
[ ] Don't know

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

[x] I downloaded it
[ ] I built it

2. If you downloaded your rom, where did you get it from?

[ ] Heads CircleCi
[ ] Purism
[x] Nitrokey (GitHub)
[ ] Somewhere else (please identify)

Please provide the release number or otherwise identify the rom downloaded 2.4 official release 3. If you built your rom, which repository:branch did you use?

[ ] Heads:Master
[ ] Other (please identify)

4. What version of coreboot did you use in building?

[ ] 4.8.1 (current default in heads:master)
[ ] 4.13
[ ] 4.14
[ ] 4.15
[ ] Other (please specify)
[ ] I don't know

5. In building the rom where did you get the blobs?

[ ] No blobs required
[ ] Provided by the company that installed Heads on the device
[ ] Extracted from a backup rom taken from this device
[ ] Extracted from another backup rom taken from another device (please identify the board model)
[ ] Extracted from the online bios using the automated tools provided in Heads
[ ] I don't know

Please describe the problem

Describe the bug I used first QubesOS 4.1.2, then 4.2 with NK Heads 2.1 for some months and never had a bad reboot; after upgrading NK Heads by internal flashing (GUI-way) to 2.4, which removes ability to reboot from within Heads, I had several bad reboots from the OS, where the laptop will shut down (looks like all the way), but then not boot up again; however, in those cases the power LED stays on, keyboard backlight can still be cycled, but pressing the power button (short press) doesn't do anything and there is nothing shown on screen, so the laptop then requires a hard poweroff (long press of power button). After that it boots again normally (with a short push of the power button). This seems to happen whenever S3 stand-by was used since last boot.

To Reproduce Steps to reproduce the behavior:

NV41 running Heads 2.4 and QubesOS 4.2:
1. Boot up
2. go to S3 stand-by
3. wake the laptop up again
4. reboot OS

Expected behavior Reboots normally always.

UndeadDevel commented 9 months ago

I just tried rebooting with 4 different kernels, including the two that I used after Heads 2.1 -> 2.4 upgrade (I definitely saw this issue with them earlier) and none of the reboots failed (20 reboots total). Just before that I did have a failure though (when rebooting from a normal use) and the failure percentage when rebooting from normal use, as said, is around 50%, so this issue seemingly can't be reproduced when just booting up -> rebooting, but only when using the system for longer and then rebooting. One day after I did the Heads upgrade there was a kernel upgrade for dom0, so theoretically this could be kernel-related, but so far I can't prove it either way.

UndeadDevel commented 9 months ago

Ok I've figured out what it's correlated to: S3 stand-by.

I can reliably reproduce the issue now by booting, going to stand-by, waking up again and then rebooting. If rebooting without stand-by in between then it reboots normally, otherwise it fails. I've updated the title and OP.

daringer commented 9 months ago

Ahh, nice thanks for the thorough investigations - seems like there is some interaction between QubesOS and the S3-sleep state. We'll will test this on a Ubuntu System to see if this is specific to QubesOS or not...

daringer commented 9 months ago

Okay, so we did some tests and it looks like the issue is reproducible with QubesOS 4.2 on the NV41, but not with Ubuntu.

@marmarek shortly searched through https://github.com/QubesOS/qubes-issues but didn't find something similar and I am not really confident that it is really a QubesOS issue - does this behavior look familiar for you?

marmarek commented 9 months ago

Currently we test only Dasharo on NV41, and I haven't seen such issue there. But, I've seen few Heads-related issues that happens depending on cold/warm reboot (on Thinkpads, not NovaCustom):

So, there are cases where some state is not fully cleared on reboot.

daringer commented 9 months ago

Thanks for the inputs @marmarek ! ok, means the next things to investigate would be the log-outputs comparison as described here: https://github.com/linuxboot/heads/issues/1557

Nitrokey / heads