search

Dasharo / dasharo-issues

The Dasharo issue tracker
https://dasharo.com/
25 stars 0 forks source link

DTS is downloaded via insecure HTTP when choosing to boot into DTS via iPXE #450

Closed zollerd closed 10 months ago

zollerd commented 1 year ago

Dasharo version

v1.4.0 https://docs.dasharo.com/variants/novacustom_nv4x_tgl/releases/#v140-2023-02-24

This is the device/firmware I observed this on, but I suspect that the issue is not specific to this device/firmware version.

Affected component(s) or functionality

Booting into 3mdeb-hosted Dasharo Tools Suite image via iPXE network boot; download of the iPXE scripts/initrd/kernel.

Brief summary

The iPXE scripts, kernel, initrd are downloaded via HTTP. This is concerning with regard to the possibility of man-in-the-middle attacks.

How reproducible

Always.

How to reproduce

See https://docs.dasharo.com/dasharo-tools-suite/documentation/#bootable-over-a-network

  1. Ensure your device has a wired internet connection
  2. Boot
  3. Open the boot menu and select the iPXE Network Boot option
  4. In the Network Boot menu, select the Dasharo Tools Suite option
  5. Observe the protocol part of the download URI on your display (reads http://)

Expected behavior

The iPXE scripts, kernel, initrd should be downloaded via HTTPS.

Actual behavior

The iPXE scripts, kernel, initrd are downloaded via HTTP.

Additional context

The EC and BIOS firmware images themselves are downloaded via HTTPS from within DTS, which is good, but doesn't help much if the DTS download can be subject to man-in-the-middle attacks.

Solutions you've tried

As an alternative to iPXE, it is possible to manually download the DTS image, write it to e.g. a thumb drive, and boot from there (see here), circumventing this issue.

miczyg1 commented 1 year ago

This is the issue I was talking about on Matrix: https://github.com/ipxe/ipxe/pull/116

Finally after YEARS of trying to merge it, it happened on 31st of March this year... https://github.com/ipxe/ipxe/pull/930

So all we need to do is to bump the iPXE revision in Dasahro builds.

macpijan commented 11 months ago

The current state of this issue is that we receive iPXE error when trying to boot from https://boot.dasharo.com https://ipxe.org/err/1c0de8

This was supposed to be fixed in the upstream iPXE here: https://github.com/ipxe/ipxe/issues/407

But the problem still persists on our side despite using this version, so we must further investigate it.

jeltsch commented 11 months ago

I must say that I’m shocked that such an obviously insecure way of starting the DTS was even considered in the first place, then was even presented as the recommended way, and finally has been in existence for years.

My impression was that the key reason for running Dasharo coreboot is having a firmware that is sort of guaranteed to be malware-free. With this insecure method of starting the DTS, a man in the middle can deliver a bogus DTS that installs malicious firmware instead of the proper one it claims to install. What advantage over having the proprietary firmware from China do we have then? Note that a single insecure boot can be enough: once there is malware in the firmware, it can make sure that all future attempts to update the firmware lead to installation of bogus firmware again.

My understanding is that PXE is supposed to be used within data centers, where all networking is under control of the operator of the computers, not across the internet.

Also note that 3mdeb may be an attractive target for a targeted man-in-the-middle attack. Among the companies that offer laptops with Dasharo coreboot installed there are those that offer laptops with special anti-tampering features. As a result, quite a few of these companies’ customers are likely to care about anti-tampering and thus be attractive as malware targets, making funneling malware into the firmware of their laptops an appealing option.

pietrushnic commented 11 months ago

I must say that I’m shocked that such an obviously insecure way of starting the DTS was even considered in the first place, then was even presented as the recommended way, and finally has been in existence for years.

We share your sentiment, too, but that is one of the problems we have to deal with, and our resources are finite. We definitely have to communicate our limitations better, so if you have ideas on how to do that, feel free to suggest your improvements.

You also know what limited resources mean.

My impression was that the key reason for running Dasharo coreboot is having a firmware that is sort of guaranteed to be malware-free.

This is an incorrect impression; no practical way to be malware-free on any modern computing platform exists. One is that you would have to have access and the ability to perform a quality audit of the whole code you are using, and there are certain pieces that are closed, and there is no plan for those being open (FSP, ME). Second is that three are boot roms built-in microcontrollers, SoCs, CPUs, and GPUs, of which code potentially consists of bugs that can lead to malware deployment.

What you can do is limit your exposure to problems. We believe open-source firmware helps in achieving that.

What advantage over having the proprietary firmware from China do we have then?

https://docs.dasharo.com/osf-trivia-list/dasharo/#what-is-dasharo

My understanding is that PXE is supposed to be used within data centers, where all networking is under control of the operator of the computers, not across the internet

PXE != iPXE

Also note that 3mdeb may be an attractive target for a targeted man-in-the-middle attack. Among the companies that offer laptops with Dasharo coreboot installed there are those that offer laptops with special anti-tampering features. As a result, quite a few of these companies' customers are likely to care about anti-tampering and thus be attractive as malware targets, making funneling malware into the firmware of their laptops an appealing option.

Agree.

From our experience, those who are very concerned can build DTS themselves from the source and use the USB method to flash. Those who are paranoid should not use DTS, but compile Dasharo and all tooling from source. This doesn't mean we neglect the problem, we still must solve this issue. Thank you for bringing it back to our attention.

jeltsch commented 11 months ago

I must say that I’m shocked that such an obviously insecure way of starting the DTS was even considered in the first place, then was even presented as the recommended way, and finally has been in existence for years.

We share your sentiment, too, but that is one of the problems we have to deal with, and our resources are finite. We definitely have to communicate our limitations better, so if you have ideas on how to do that, feel free to suggest your improvements.

I understand your resource limitations, but I’m not calling for new features, like iPXE boot with SSL. In fact, I’m calling for less features.

In my opinion, booting the DTS over an insecure connection should never have been implemented in the first place. The approach of downloading the DTS, verifying a cryptographic hash of it, placing it on a USB memory stick, and then booting from this stick is perfectly fine. It’s analogous to what Linux users have been doing for a long time when installing new Linux distributions. I guess that people who go so far as to seek to install special firmware shouldn’t be scared off by the slight complication this process brings over an iPXE-based solution.

So I’d argue that booting the DTS via USB memory stick should be the only option offered. At least, the solution with the insecure iPXE boot shouldn’t be recommended but rather should come with a warning sign, pointing out the insecurity.

My impression was that the key reason for running Dasharo coreboot is having a firmware that is sort of guaranteed to be malware-free.

This is an incorrect impression; no practical way to be malware-free on any modern computing platform exists.

Yes, unfortunately. However, it should at least be reasonably sure that the UEFI firmware that a machine runs is the one that the user intended to run.

What advantage over having the proprietary firmware from China do we have then?

https://docs.dasharo.com/osf-trivia-list/dasharo/#what-is-dasharo

Well, all these things only apply if the firmware ending up in the device is really the Dasharo firmware, but this is exactly what is put into question by the use of unsecured internet transfer of the DTS.

My understanding is that PXE is supposed to be used within data centers, where all networking is under control of the operator of the computers, not across the internet

PXE != iPXE

I think that my point remains: Network boot without secured connections is only safe if the whole network between the client and the server is under the control of the operator of the client and thus can be trusted.

From our experience, those who are very concerned can build DTS themselves from the source and use the USB method to flash.

I would be entirely happy to use a pre-built DTS from a USB memory stick. Of course, I could re-install the firmware using this configuration, but, if the existing firmware has already been tampered with by a previous iPXE-based installation, my approach wouldn’t be safe either.

pietrushnic commented 11 months ago

In my opinion, booting the DTS over an insecure connection should never have been implemented in the first place.

Is there anyone who disagrees with you? We can't time travel, so we have to live with our mistakes, which we regret, and we will try our best to improve things in the future. Is there anything else we can do?

So I’d argue that booting the DTS via USB memory stick should be the only option offered. At least, the solution with the insecure iPXE boot shouldn’t be recommended but rather should come with a warning sign, pointing out the insecurity.

Please note documentation is open-source, and contribution is very welcome. Pointing to issues is important, but someone has to take action and fix what can be fixed. Action is worth way more than words. Is there any specific location in documentation, specific information, or phrasing you would like to see?

Feel free to review my proposition for modification: https://github.com/Dasharo/docs/pull/727

jeltsch commented 11 months ago

In my opinion, booting the DTS over an insecure connection should never have been implemented in the first place.

Is there anyone who disagrees with you?

My impression was that my opinion on this matter was not a consensus. Good if it is.

We can't time travel, so we have to live with our mistakes, which we regret, and we will try our best to improve things in the future. Is there anything else we can do?

You can remove the feature. 🙂

So I’d argue that booting the DTS via USB memory stick should be the only option offered. At least, the solution with the insecure iPXE boot shouldn’t be recommended but rather should come with a warning sign, pointing out the insecurity.

Please note documentation is open-source, and contribution is very welcome. Pointing to issues is important, but someone has to take action and fix what can be fixed. Action is worth way more than words.

Well, pointing out issues is already action, and people involved in the project can usually fix things more easily than outsiders like me.

Is there any specific location in documentation, specific information, or phrasing you would like to see?

Feel free to review my proposition for modification: Dasharo/docs#727

I just read through it, and to me it looks like a great change, as the added text quite strongly discourages the use of iPXE and justifies this discouragement. Thank you!

marmarek commented 11 months ago

Related (a duplicate?) issue: https://github.com/Dasharo/dasharo-issues/issues/270 While HTTPS is desirable anyway, some integrity protection can be provided by signatures on the boot binaries (but it won't protect against rollback attack for example).

wessel-novacustom commented 11 months ago

This way of downloading DTS is not only insecure for our customers because of possible MitM attacks, it also harms NovaCustom's and Dasharo's reputation.

@pietrushnic / @macpijan Let's make this issue a no. 1 priority for further Dasharo and DTS releases next year. As security is top priority for NovaCustom, we would like the development costs to realise the two points below to be assigned to us.

  1. Verification of the downloaded binaries: https://www.ipxe.net/cmd/imgverify
  2. Booting DTS over HTTPS: https://ipxe.org/crypto

I will open a Jira ticket for this.

First thing that needs to be fixed right now is that the latest version of Dasharo Tools Suite (v1.2.11 as we speak) should be added on the release page. @macpijan

Also, the firmware update page needs a statement about this issue. At least a link to this issue should be present there. @macpijan

EDIT (21 December 2023 12:02 PM CET):

As the DTS releases page has been updated, one can now safely flash offline by using a USB pen drive. So the requirement of external flashing is no longer relevant. To be able to read it back, though, I didn't remove what I wrote earlier today, but quoted it:

@jeltsch Probably the safest way now to update the firmware is by external flashing with another (Linux) computer. Anyone who is concerned about this issue can get an external firmware programmer free of charge by contacting us (please mention your order number and current address). If you are not a NovaCustom customer, you can buy an external programmer here. We highly recommend our 3.3V programmer. Other CH341A programmers do their work at 5V, which can harm the SPI chip.

You can download the latest public binary on the Releases page. For NovaCustom laptops, you can find this under https://docs.dasharo.com/unified/novacustom/overview/ --> your device --> Releases.

To verify the binary integrity with the hash and the signature, please follow the instructions in Dasharo release signature verification using this key.

To flash a (public) binary externally, you need flashrom version v1.3+. By default, the most popular operating systems like Ubuntu install version v1.2 when using apt. So you need to build flashrom from source to get a higher version, or you need to contact us (or your reseller) to get the full binary --> to flash the whole main SPI chip including the Intel ME region.

sudo apt install git make binutils build-essential ca-certificates libpci-dev libftdi-dev libusb-1.0-0-dev git clone https://github.com/flashrom/flashrom You can disconnect from the internet at this point. cd flashrom make CONFIG_CH341A_SPI=yes sudo make install To flash your verified binary, connect your programmer to the computer you are using for external flashing and the WSON 8x6 probe to the motherboard. Then execute this: flashrom -p ch341a_spi --ifd -i bios -w /path/to/your/verified/binary.rom

Obviously, /path/to/your/verified/binary.rom needs to be replaced by the actual path of your verified binary ROM file.

The flasing process can take up to 14 minutes. Once the process is done, the output will include the string VERIFIED. You can then remove the programmer.

In case flashrom finds multiple chips, you can just pick the first one.

For instance, the output might be:

Found Winbond flash chip "W25Q256FV" (32768 kB, SPI) on ch341a_spi. Found Winbond flash chip "W25Q256JV_Q" (32768 kB, SPI) on ch341a_spi. Multiple flash chip definitions match the detected chip(s): "W25Q256FV", "W25Q256JV_Q" Please specify which chip definition to use with the -c option.

So in that case, you need to run flashrom with the first detected chip by using the -c flag:

flashrom -p ch341a_spi --ifd -i bios -w /path/to/your/verified/binary.rom -c W25Q256FV

Last but not least: especially interesting for those who do not have technical knowledge or time, you can apply for warranty free of charge so we can take care of this external flash operation. If you have chosen our physical anti-tamper feature, we will replace the anti-tamper sealing and re-seal the screws and the shipping boxes, plus send you the new photographs of the screw sealing via Proton Mail. If there was paid for this initially, this will be free of charge.

pietrushnic commented 11 months ago

@marmarek definitely duplicate.

macpijan commented 11 months ago

The problem with HTTPS is related to the Let's Encrypt certificates and known iPXE problem as reported here: https://github.com/ipxe/ipxe/issues/606#issuecomment-1865945119

We are looking for a workaround to this problem.

macpijan commented 11 months ago

@marmarek definitely duplicate.

Marek also mentions imgverify, so here we can focus on HTTPS, in Marek's issue we can discuss additional signature verification mechanisms.

macpijan commented 11 months ago

We have tested one "workaround" by adding the Root X1 as an additional trusted CA to our iPXE build and in this case HTTPS to boot.dasharo.com works fine.

Tomorrow we will also try out a server solution, to reduce the amount of changes on the client side. Based on the results, we will apply one of the two and include it for the releases happening in the near future.

mindaugas147 commented 10 months ago

@jeltsch thank you for your thoughtful issue https://github.com/Dasharo/dasharo-issues/issues/642 and comments, it brought change.

My impression was that my opinion on this matter was not a consensus. Good if it is.

It was not a consensus. Linking an example issue how it was handled before: https://github.com/Dasharo/dasharo-issues/issues/456

I was rolling my eyes as well during my recent (first) firmware upgrade, seeing: 1) http 2) failed checksum before the upgrade 3) blue screen verification error after the upgrade (https://github.com/Dasharo/dasharo-issues/issues/556). All of this in a single update, but in the end, I trusted dasharo team know what they are doing, and did not followed up much on this particular issue.

Great that people like you @jeltsch exist here, leveling dasharo team up to a higher security standard.

I guess that is another plus for open source firmware.

Let's make this issue a no. 1 priority for further Dasharo and DTS releases next year.

Thank you @wessel-novacustom for great customer care.

jeltsch commented 10 months ago

Thanks a lot for your feedback, @mindaugas147. I’m happy that I could be of help not only for myself but also others. :slightly_smiling_face:

miczyg1 commented 10 months ago

We have added information to the documentation about the issue: https://docs.dasharo.com/dasharo-tools-suite/documentation/#temporary-suspension-of-network-booting-for-dts

wessel-novacustom commented 10 months ago

@miczyg1 Not to forget to mention that DTS is now downloaded via HTTPS.

@ anyone reading this: Please note that, for NovaCustom laptops with 11th Generation Intel Core processor with version v1.5.1 and lower, for NovaCustom laptops with 12th Generation Intel Core processor with version v1.7.1 and lower, and possibly for other Dasharo hardware, your firmware might initially still point to HTTP at first, which is then redirected to HTTPS. To solve this, you will need to upgrade DTS to a newer version. @macpijan please correct me if I'm wrong.

@zollerd I think this issue can be closed now.

miczyg1 commented 10 months ago

Not to forget to mention that DTS is now downloaded via HTTPS.

It is mentioned in the link

macpijan commented 10 months ago

@wessel-novacustom

To solve this, you will need to upgrade DTS to a newer version. @macpijan please correct me if I'm wrong.

Firmware version is critical, not DTS version, as the link is embedded into firmware. The firmware versions supporting HTTPS are listed in link provided by @miczyg1 : https://github.com/Dasharo/dasharo-issues/issues/450#issuecomment-1876928417

wessel-novacustom commented 10 months ago

@macpijan Thank you for this addition.

wessel-novacustom commented 10 months ago

@zollerd The issue should have been fixed now starting from the new firmware versions that has been released today and earlier this month. Can the issue be closed or is it still relevant?

BeataZdunczyk commented 10 months ago

@wessel-novacustom I am removing the NovaCustom labels, as all new releases (NovaCustom 12th Gen 1.7.2, 11th Gen 1.5.2) include this fix. Let's keep this issue open until MSI releases are published, which will happen shortly.

zollerd commented 10 months ago

Thank you all for giving this more weight and thank you 3mdeb for fixing this.

I'm now on v1.5.2 on my NV4x 11th Gen, but I can't test the fix at the moment - if i choose "Dasharo Tools Suite" in Dasharo Network Boot Menu, I'm always thrown back to the Network Boot Menu after about 2 seconds. It's not clear to me why this happens because I made sure that all prerequisites are met.

It wouldn't hurt to get independent confirmation that this is indeed fixed before closing this, but I assume you did. In case I can get it to work I'll report back, but as far as I'm concerned, feel free to close the issue once you see fit.

wessel-novacustom commented 10 months ago

Hello @zollerd ,

Thank you for getting back here.

I'm now on v1.5.2 on my NV4x 11th Gen, but I can't test the fix at the moment - if i choose "Dasharo Tools Suite" in Dasharo Network Boot Menu, I'm always thrown back to the Network Boot Menu after about 2 seconds. It's not clear to me why this happens because I made sure that all prerequisites are met.

It wouldn't hurt to get independent confirmation that this is indeed fixed before closing this, but I assume you did. In case I can get it to work I'll report back, but as far as I'm concerned, feel free to close the issue once you see fit.

BeataZdunczyk commented 10 months ago

Closing this issue as we have just published release for MSI boards, which also include the HTTPS fix.