After firmware update, error message with blue screen report that firmware signature verification failed..but system works perfect.

[sateuwdie]

Dasharo version 1.7.0

Dasharo variant Dasharo (coreboot+UEFI) v1.7.0

Question text Today I have upgraded the firmware via ipxe from dasharo tools suite, version: 1.7.0 The update apply but the system poweroff immedately during update (of course was connected to AC source) i press the power button and system come up..with a blue screen tell me firmware was not updated correctly. The system boot, all works, no error reported, I can enter in coreboot menu, etc. Can I stay safe? Thanks The system is NS5x_NS7xPU

[wessel-novacustom]

Thank you for your report!

The immediate power off behaviour is normal as the EC firmware needed an update as well, this cannot be prevented.

We asked our team to stop the updates from being public, as there are some critical issues with the latest versions.

Nevertheless, I believe it can be ignored safely, as this is probably a result of the entire BIOS region being flashed. I also believe that the blue screen you saw was a message that the firmware could not verify its integrity because of a newer EDK II version. This hypothesis hasn't been verified by the developers yet, though.

Please contact us directly in case you need instructions to downgrade to v1.5.0.

If you do not experience any critical issues with v1.7.0, you can safely continue to use it. A hotfix release (v1.7.1) will be published very soon.

[macpijan]

@sateuwdie Thanks for the report.

Can you please clarify a few more things:

1. What is your current firmware version? Does it show 1.7.0 right now?
2. Does this blue popup shows up every time, or it was just one time?
3. Do you have a photo or remember the error code (recovery reason) from the popup? Was it like in the attached picture, or something else?
4. What is your the firmware version before you have upgraded to 1.7.0?

[sateuwdie]

Thanks for answer. I reply to all

a) No critical issues

b)Yes, the blue screen image was this

and it show it only one time after firmware-upgrade.

c)the previous firmware version was 1.6.0 now report correct version 1.7.0

[wessel-novacustom]

I got the same message when upgrading from v1.6.0 to v1.7.1.

[macpijan]

@wessel-novacustom Did you have the same exact 0x5F error as on the screen above?

[macpijan]

Some ideas from discussions so far:

• we inform in this message that we have booted from the recovery partition
• we might display this message if any error code appears, but maybe not all of them means the recovery request has happened?
• we need to have a closer look on the implementation details here.

[pietrushnic]

@macpijan related issues

• https://github.com/Dasharo/dasharo-issues/issues/489
• https://github.com/Dasharo/dasharo-issues/issues/442
• https://github.com/Dasharo/dasharo-issues/issues/269

There are a number of issues regarding this feature, IMHO, which means there may be some design flaws. What bothers me is the fact that we easily display messages to scare users, but we do not provide enough guidance on the real meaning of the problem and recovery mechanics.

So, we implemented a protection mechanism with poor detection quality (many false positives) and no recovery. Every design of security features should treat all stages of protection, detection, and recovery equally seriously.

If, for some reason, we have a problem identifying why we have an issue, we at least should give some diagnostics tools or procedures to users so they know what they should do in such a situation.

[wessel-novacustom]

I just reproduced this issue on the NS51PU-i7 after flashing v1.7.1

[mkopec]

I believe if we did updates only for the RW_SECTION_A/B regions, the issue would no longer apply. Updating the RO partition will cause problems such as these. It also makes sense from the security perspective and would allow us to use the SPI chip's flash protection.

This is a big advantage that Vboot gives us and we're not making use of it.

[wessel-novacustom]

@macpijan I think we should try that tomorrow.

[macpijan]

I believe if we did updates only for the RW_SECTION_A/B regions, the issue would no longer apply.

We know that already, but this is not a solution to this particular problem.

Updating the RO partition will cause problems such as these.

Still, we need more in-depth understanding of this particular problem.

This is a big advantage that Vboot gives us and we're not making use of it.

We already take advantage of vboot, and try to minimize the necessity of updating RO partition. Here we had very good reasons to do so (like using Dasharo vboot keys), and you know in reality we may face others - like the necessity to extend the flash layout to fit 2 FSP copies, etc.

[macpijan]

detection quality (many false positives) and no recovery

This is not really proven yet. Maybe we do in fact boot from recovery in these cases, so the message would be true. We need to have a closer look at the case when reproduction happens.

[pietrushnic]

This is not really proven yet. Maybe we do in fact boot from recovery in these cases, so the message would be true. We need to have a closer look at the case when reproduction happens.

@macpijan Yes, the technical root cause is not yet proven, and I can't comment on the reason in the code because I don't know this implementation so well. My comment is purely how it seems from a UX perspective. We need better tests of this feature.

[miczyg1]

@macpijan upgrading from v1.7.0 to v.1.7.1 does not reproduce this issue. How to get DTS image which upgrades from v1.5.0 to v1.7.0? My guess is the DTS v1.2.10 but the files are nowhere to be found. I would like to use the same procedure when reproducing it, especially that the error occurred only once for unknown reason (the hash space lock was in v1.6.0 too).

[macpijan]

@miczyg1 Wessel has been able to reproduce it going from 1.6.0 to 1.7.1 with the DTS available in iPXE. It does not happen always, though...

It does not matter if we are going to 1.7.0 or 1.7.1 from 1.6.0. In both cases the bios region gets flashed.

especially that the error occurred only once for unknown reason

This error has happened multiple times already, for Marek, Wessel, and various end users.

[miczyg1]

This error has happened multiple times already, for Marek, Wessel, and various end users.

I meant the error occurs only once after flashing.

So all it takes is to flash v1.6.0 back and update from DTS again

[miczyg1]

v1.6.0 -> v.1.7.1 still no reproduction.

[miczyg1]

Found possible culprit: https://github.com/Dasharo/coreboot/blob/dasharo/src/mainboard/clevo/adl-p/cmos.layout#L22 perfectly overlaps with https://github.com/Dasharo/coreboot/blob/dasharo/src/soc/intel/common/basecode/ramtop/ramtop.c#L35

Having such luck is beyond coincidence...

[miczyg1]

https://github.com/Dasharo/coreboot/pull/434

[wessel-novacustom]

I will need to test this on a brand new device that has never had Dasharo installed before. For example, by installing v1.6.0 on NS5x 12th Gen, install the operating system and use it normally. Then I will install the test binary with the command flashrom -p internal --ifd -i bios -w /path/to/coreboot.rom to see if I get the message again.

[pietrushnic]

Another instance of this problem was reported on Dasharo - Support channel. Fixing it is one thing, but correctly communicating such a scary message may pop up is another. IT would be great if we can address both, because this bug really damage Novacustom and Dasharo reputation.

[sycam0r-e]

Another instance of this problem was reported on Dasharo - Support channel

Yes, that was me. :)

I am a relatively naïve non-techie user who updated firmware for the very first time so allow me to give respective UX feedback.

What could be (or could have been) better:

• The error message itself should be more descriptive and clearer.
  • A "signature mismatch" is itself not informative enough for the average user. Try to elevate this on a descriptive level that abstracts away from the technical level; if there is a signature mismatch then the "authenticity / trusthworthiness of the firmware could not be established" might be more comprehensible for the average user?
  • Get rid of language ambiguities as they lead to insecurities for the user and might lead the user to just discard the message (such as the word "likely").
  • The implications are not clear; how exactly should a user do what? How can one "verify [...] the proper source"?
• The power off caught me by surprise and scared me a lot. I wasn't aware that EC can/will be updated concurrently. Document that: a) EC can/will be updated concurrently and b) the immediate power off behaviour after EC firmware, -> both, somewhere on the BIOS firmware update pages as these are linked in the firmware update emails.
• As far as I understand your current thinking is that the warning message being shown is erroneous and that users have actually nothing to worry about. However, this is not stated clearly in this issue. Make a separate comment in this issue detailing under which conditions (e.g. if the iPXE boot menu has been used) users do not have to worry.

I hope this is informative for you, I'm just trying to help since I wholeheartedly support what you all do! 👍

I noticed some other things I'd desire regarding the general documentation of a firmware update, but this would be offtopic here. Let me know if and where I should comment on that. :)

[wessel-novacustom]

@sycam0r-e Thank you for your valuable feedback! I hope @macpijan can pick this up and improve a bit were needed/agreed.

I noticed some other things I'd desire regarding the general documentation of a firmware update, but this would be offtopic here. Let me know if and where I should comment on that. :)

• I'm very curious, please let me know ;-)

[BeataZdunczyk]

Your insights are highly appreciated, @sycam0r-e, thank you for your contributions!

Let me know if and where I should comment on that. :)

Please consider creating issues on our Dasharo Issues page.

You can conveniently create new issues using this link: Create a New Issue. Providing detailed context and information, including screenshots or photos, will help us address and resolve the issues effectively.

Feel free to join our Matrix community: https://matrix.to/#/#dasharo-laptops:matrix.org to share your thoughts.

[mkopec]

Resolved by commit https://github.com/Dasharo/coreboot/pull/437/commits/f71119b07e6c17b5f1b4077fd7342b110017d5f1

[wessel-novacustom]

@mkopec Has this been verified? I can imagine it is hard to reproduce ...

[sycam0r-e]

Allow me to direct your attention to one of the more important parts of my earlier message that likely just got lost:

As far as I understand your current thinking is that the warning message being shown is erroneous and that users have actually nothing to worry about. However, this is not stated clearly in this issue. Make a separate comment in this issue detailing under which conditions (e.g. if the iPXE boot menu has been used) users do not have to worry.

Emphasis added.

[wessel-novacustom]

@macpijan This issue hasn't been resolved yet. It was closed, but can be opened again.

The thing is that it only seems to happen once per device when upgrading from version < v1.7.0 to version >= v1.7.0. In our production, we always apply version v1.6.0 and then upgrade to version v1.7.2. In all cases, we still get this error, even though we use the latest version of Dasharo Tools Suite. Do you have any ideas how we can solve this issue? NovaCustom users report a lot that they are scared of the error message.

[macpijan]

Issue has been reopened, as it seems the problem has not been solved, and it still appears in the following scenario:

1. Get a new device with proprietary firmware
2. Flash Dasharo 1.6.0 (or older)
3. Update to Dasharo 1.7.2
4. The popup appears on the first boot.
5. This problem cannot be reproduced on the same hardware unit again thus far, making it very tricky to diagnose

In: https://github.com/Dasharo/dasharo-issues/issues/556#issuecomment-1809940586 there was a potential fix, but it seems that the problem still appears on the new hardware units.

[wessel-novacustom]

@macpijan Exactly.

As we have new hardware all the time, we can do testing for this issue.

[marmarek]

I believe I hit this issue on MSI Z690 too. Updated from 1.1.1 to 1.1.3 using flashrom -p internal -w [path] --fmap -i RW_SECTION_A -i RW_SECTION_B, then typed "reboot" and got message like this:

After pressing ENTER, I immediately entered setup, and was greeted with 1.1.1 version there.

[mkopec]

@marmarek I believe the command for updating to v1.1.3 should be flashrom -p internal --ifd -i bios according to the DTS scripts: https://github.com/Dasharo/meta-dts/blob/017ec32aab3f4481c3a904d507212fcf73570cc7/meta-dts-distro/recipes-dts/dts/dts/dts-functions.sh#L323

The firmware update docs need to be updated to reflect this: https://docs.dasharo.com/unified/msi/firmware-update

[marmarek]

hmm, maybe better describe it on https://docs.dasharo.com/unified/msi/firmware-update/ then? the section "Version v1.1.0 or newer" says to do just RWSECTION{A,B} (there is a note about 1.1.2 too, but it's unclear...)

[mkopec]

Yes, the docs appear to be outdated, see my edit above. Need to update the docs so that the entire BIOS region is flashed.

[marmarek]

ok, after updating entire BIOS region, correct version is running, but still got the message after update.

[mkopec]

The error indicates that Vboot could not save the hash of recovery MRC cache to TPM NVRAM. Since this error appears once per machine lifetime (after switching from Insyde), it may mean that we need to do a reset of the TPM during initial deployment. For upcoming releases we may work around this issue by turning on secdata mocking.

@miczyg1 wdyt?

[mkopec]

We've sent binaries for testing to @wessel-novacustom , please let us know when you're able to confirm that the issue is fixed now.

[wessel-novacustom]

We've sent binaries for testing to @wessel-novacustom , please let us know when you're able to confirm that the issue is fixed now.

I'll do tomorrow at latest.

[wessel-novacustom]

I have tested the binary on an NV41PZ running Dasharo v1.6.0 (never been to v1.7.x on that unit before) with the following DTS commands:

wget http://192.168.1.75/coreboot.rom (this is from a local server of course) sha256sum coreboot.rom says: ec05de09b986b45fb0d1e77262777c95a5a4f7f1ea58952dcbab30c0daf09bb6 coreboot.rom I have flashed the binary with the following command: flashrom -p internal --ifd -i bios -w coreboot.rom && systemctl poweroff After a few minutes, the laptop has shut down.

I powered it on and the message did not appear as it usually does after an update.

I have tried this on multiple devices. With both EC of version v1.6.0 and EC of version v1.7.2, with the same results.

I have even updated manually from v1.6.0 to the original v1.7.2 version on an NV41-ADL that again never had Dasharo v1.7.x before. I did this to make sure that the DTS migration process of the original update method isn't causing the issue. The pop up showed up at the first boot, which is not the case for the test binary.

So yes, we have the sweet spot, it seems.

[mkopec]

Closing, as the issue is confirmed to be fixed. I also posted some info in a related issue: https://github.com/Dasharo/dasharo-issues/issues/689#issuecomment-1981496430

[baptx]

@wessel-novacustom @mkopec It looks like the issue is not fixed yet because today I updated the firmware from version 1.6.0 to 1.7.2 on my NV41 Series laptop and the blue warning appeared once.

[wessel-novacustom]

@wessel-novacustom @mkopec It looks like the issue is not fixed yet because today I updated the firmware from version 1.6.0 to 1.7.2 on my NV41 Series laptop and the blue warning appeared once.

It has been fixed for new releases, but there hasn't been any new stable version released yet.

Maybe @mkopec or @macpijan can confirm.