EEPROM write protection

[rtfmoz2]

Security: The Boot Process -- Forum Discussion

".. it is possible to physically write-protect both EEPROMs via a simple resistor change on the board. Details will be published in the schematics." -- Documentation

Not yet indicated in the schematics. Please add this information.

[meeDamian]

Note that supply chain attacks exist, meaning EEPROM can already be compromised upon delivery.

Obviously, the HW itself can be compromised as well, but EEPROM is a particularly easy way to do it, as it's software only, and compromising it could take literal seconds.

[JamesH65]

@jimbojr Can we add this resistor mod to the schematics?

[mAPBhlJ]

Note that supply chain attacks exist, meaning EEPROM can already be compromised upon delivery.

By that logic the Broadcom chip on the Raspberry Pi could be compromised as well and have a coprocessor that does whatever. If you feel it is required; you can re-write the EEPROM yourself upon arrival of your Raspberry Pi (if you trust the Broadcom unit).

Obviously, the HW itself can be compromised as well, but EEPROM is a particularly easy way to do it, as it's software only, and compromising it could take literal seconds.

Yes, the EEPROM can even be overwritten by software, altering your bootcode. The point is; this resistor modification should make this impossible in software.

In this case; someone can still walk up to your Raspberry Pi and put the resistor back on, or solder a completely different EEPROM on it, but that's not the point.

Anyways.

The documentation states SPI EEPROM, which would mean an external EEPROM module.

These modules often have an "write enable" input pin, by cutting the trace to this input pin (or removing the resistor that's on this line), you can make this EEPROM essentially a (EE?)ROM ;)

I would say it's the black chip on the back labeled "SKUK" since it's close to the SD Card interface (which commonly uses SPI). The label "SKUK" doesn't help me on this and I'm not going to try cutting those traces one-by-one to identify the pins, nor do I have the time to reverse engineer them using an oscilloscope, but be my guest.

You may try to reverse engineer the recovery .bin files to see what it does to write to it; but I'm not even sure if that's viable.

Long story short: We'll have to wait for the Raspberry Pi org to release the EEPROM part of the schematics or simply tell us which trace/resistor to cut/remove.

[JamesH65]

@jimbojr @mike-stimson Is this still on the cards for improvement?

[JamesH65]

@Roger-Thornton Added to CC.

[stiell]

Decided to look a little into this myself.

According to https://hackaday.io/page/6372-raspberry-pi-4-boot-sequence, talking about the boot EEPROM:

This is a standard Winbond W25X40 chip, access through SPI0 on GPIO 4043.

Datasheet: https://www.winbond.com/resource-files/w25x40cl_e01.pdf

It's an 8-pin chip. There are two 8-pin chips on the board, both on the front near the A/V jack. Size is 2x3mm. This corresponds to the UX package type of the W25X40 chip. Top side marking according to datasheet is:

4Hxxx 0Gxxx

This matches the marking on the chip closest to the A/V jack.

The other chip near the A/V jack is a Winbond W25X10 EEPROM chip connected to the VL805 (USB controller) chip, so I presume the W25X10 contains the USB controller firmware.

"/WP" (Write Protect Input, active low) is on pin 3 according to the datasheet. Both chips' "/WP" pins are connected together and to TP5 on the back of the board, as well as a 1k8 resistor next to TP5. The resistor goes to 3V3. Next to it is a pair of unpopulated pads that allow for a resistor between TP5 and GND.

Moving the resistor to these unpopulated pads, or placing a 0-ohm resistor there, should pull "/WP" down and activate the pin. However, the BP2/BP1 (Block Protect) and SRP (Status Register Protect) bits in the chip's status register must also be set to 1 (factory default is 0) for write protection to be enabled. I don't know if BP2/BP1 and SRP already gets set to 1 in either chip.

(Edited to mention BP2/BP1.)

[stiell]

I played around a little with spidev and strace + vl805 (is there any source code for the vl805 binary?), and found that neither EEPROM chip has its protection bits set.

So for write protection to work, something has to be done on the software side as well. This should be reasonably simple to do for the boot EEPROM. Not sure about the VL805 EEPROM, but the vl805 binary is at least able to talk SPI to the EEPROM in a slightly more convoluted way (confirmed using strace), so it should be possible.

[lurch]

is there any source code for the vl805 binary?

https://github.com/raspberrypi/rpi-eeprom/commit/e35f0c7d41ae88f43c48c29244e24466bdbc101d (in combination with https://github.com/raspberrypi/rpi-eeprom/blob/master/LICENSE ) suggests not :wink:

[ejolson2005]

Is there any progress on this issue? I have a use case at

https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=275911&p=1672167#p1672167

where being able to write-protect the EEPROMs is almost essential. Thanks in advance for everyone who has looked into this and is presently working on it.

[cleverca22]

(assuming OTP66 isnt modified*)

you can just put an uSD card with recovery.bin to reflash the eeprom at any time, even if the firmware is malicious

the mask rom gives recovery.bin priority, and will never even look at the eeprom

*: OTP66 lets you configure a gpio to be able to disable SD or SPI boot, so you could hold a button to skip one of them

[lurch]

Yeah, but inserting a uSD card requires physical access, and having physical access (nearly) always trumps any security measures?

[rtfmoz2]

That's right, you can't protect the security of a device if they have physical access. We are trying to protect against remote attempts at compromising the device and reprogramming it for alternative purposes.

On Thu, 4 Jun 2020 at 21:29, Andrew Scheller notifications@github.com wrote:

Yeah, but inserting a uSD card requires physical access, and having physical access (nearly) always trumps any security measures?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/documentation/issues/1302#issuecomment-638790863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJYYSIVEY72KEQUKD7YOIV3RU6AS5ANCNFSM4I6YVWGA .

[cleverca22]

i was refering to recovery.bin more in the sense that the owner can undo most supply-chain attacks, once he gets physical access (assuming OTP66 hasnt been messed with)

and the pi4 eeprom doesnt have enough storage for 2 copies of the eeprom, so such malware would have a very difficult time keeping a copy of the eeprom over reboot, to fake having flashed and fool you

he can also use that to undo any remote attacks, after they have been discovered

[lurch]

i was refering to recovery.bin more in the sense that the owner can undo most supply-chain attacks

@cleverca22 Ahhhh, I see. It looked like you were replying to @ejolson2005 rather than @meeDamian :wink:

and the pi4 eeprom doesnt have enough storage for 2 copies of the eeprom, so such malware would have a very difficult time keeping a copy of the eeprom over reboot, to fake having flashed and fool you

As you already noted, "the mask rom gives recovery.bin priority, and will never even look at the eeprom" so even if the eeprom was big enough to store two copies of the bootloader, there'd be no way for a "malware" to stop itself being overwritten by recovery.bin? (at least AIUI). Which of course is all hypothetical anyway :slightly_smiling_face:

[ejolson2005]

i was refering to recovery.bin more in the sense that the owner can undo most supply-chain attacks, once he gets physical access (assuming OTP66 hasnt been messed with)

I think my usage case may be worth describing as it clears up confusion about physical access. Imagine this: A rack of about 1000 colocated Pi computers at a high security data center that has their own police and fire department. These Pi computers have no SD cards and have been configured to network boot. They are rented out to tenants who can reboot them using a web interface that toggles the run pin to load whatever image is selected--sort of like EC2 or VMware, except with real physical hardware to avoid the side-channel leakage of information that keep making security headlines and other difficulties.

At any rate, the tenants have full control of the Pi once it's booted and can do any root thing people do with stuff in a cloud. Tenants will come and go. Between them, it is important to clean up the Pi. In particular, it would be difficult to rent the Pi out if it no longer network boots.

While the tenant did not have physical access, the person cleaning the Pi between tenants does not readily have physical access either. It simply does not scale well to go to the data center, pass through security and insert the firmware reset SD card into the corresponding Pi between each tenant. Fortunately, such efforts would not be needed if after the network boot option was set, it couldn't be changed.

[lurch]

AIUI, if you've got FREEZE_VERSION=1 and ENABLE_SELF_UPDATE=0 in your eeprom config, and there's no physical SD card in the Pi (so recovery.bin can't be used), then the "standard" rpi-eeprom-update methods won't allow the eeprom to be altered. See my long comment here. But if the tenants have full root access, then I guess they could theoretically use the unsupported flashrom mechanism for updating the eeprom on a network-booted Pi?

[ejolson2005]

But if the tenants have full root access, then I guess they could theoretically use the unsupported flashrom mechanism for updating the eeprom on a network-booted Pi?

That's right.

While FREEZE_VERSION=1 and ENABLE_SELF_UPDATE=0 prevent accidental changes, they do nothing to avoid a malicious change. If one is colocating in a data center with crazy amounts of physical security while guaranteeing full protection from side-channel information leakage by renting simple single-tenant non-virtualised hardware one attracts two types: Those who need security and those attracted by the data of those who need the security.

That second group includes well funded criminals and spy agencies expert in advanced persistence threats. Thus, for this application, one needs a physical way to disable changing the firmware or any of its settings. The existence of such a mechanism is claimed by the documentation and clearly present in the hardware, but it is not properly described and the needed software tools to make it work are missing.

[JamesH65]

@jimbojr @Roger-Thornton We could really do with some clarity on this. Does the resister change still exist? Are we planning to document it? @timg236 Are the security issues claimed above valid? Can anyone create an arbitrary boot loader and use that to break in to 'something'?

[timg236]

TLDR, there is no signed boot on Pi 4 B, move along nothing to see here

The 'register' is documented in the public datasheet for Winbond and anyone is free to go and patch flashrom to configure it. I think Google might have had a fork which did that.

If set it will break rpi-eeprom-update until flashrom is set to call it, so there's little motivation to wrap that into a standard tool.

Kernel and start.elf run with full privileges so you can get equivalent access by just modifying the file-system. You don't gain anything by running stuff as the bootloader. There's a slight gap for the 'pi' user which has access to GPIO but clearly web-services etc shouldn't be run as 'pi'.

[ejolson2005]

Kernel and start.elf run with full privileges so you can get equivalent access by just modifying the file-system. You don't gain anything by running stuff as the bootloader.

If booting from an SD card it is possible to verify the card in a separate system and even write protect it, if necessary. If the Pi is network booting from a secure file server (or through rpiboot), it is possible to check the integrity of the boot files remotely from the server.

The firmware on the Pi can not be checked in this way. Even if the firmware is authentic, just changing the flags to disable network boot is enough to cause havoc from a remote administration point of view.

On the other hand, write protecting the firmware and firmware settings makes it possible to reset the Pi to a known good state just by power cycling it. In my opinion, this opens up the Pi for use as a single-tenant bare hardware cloud solution in a way that could scale.

If set it will break rpi-eeprom-update until flashrom is set to call it, so there's little motivation to wrap that into a standard tool.

I'm quite disappointed that the documentation was recently changed to remove mention of the ability write protect the firmware.

https://github.com/raspberrypi/documentation/commit/ada44909b09d676e42d9404ef318ff551ad5dfbc#diff-06e6ce58974db32aa68493c23e751e46

That wasn't the response I was expecting. You mention there is, in fact, a tool written by Google to flip the needed bit. Where can one get that tool?

[rtfmoz2]

So to summarise, the official response to this issue is to delete the reference to write protect in the documentation?

I look forward to hearing the internal comments on this.

[timg236]

The document you are referring to high-level usage guide and following the tidy up to make this easier to understand. Anything which involves modifying the board (which will probably invalidate the warranty), and rolling your own version of flashrom has no place in such a document. This is a niche issue on Model B and not supported. If it ever were to be supported on a board which didn't require a hardware mod then there would be a proper document.

[ejolson2005]

This is a niche issue on Model B and not supported.

Given current trends, "desktop replacement" is likely a much smaller niche than cloud. Upon noting Google's interest in it, being able to write protect the firmware is a big selling point (not only for the compute module) and should, in my opinion, be mentioned in the feature summary as well as documentation.

The original documentation said "However, it is possible to physically write-protect both EEPROMs via a simple resistor change on the board. Details will be published in the schematics." The use of the word simple made me think this would be simple rather than not supported. That original document would have been quite easy to understand if details of how to write protect the firmware had actually been provided in the schematics as claimed.

Moving forward, it would be great if a more technical document were produced that describes how to write protect the firmware and firmware settings. I would be happy to look that over, try the procedure and report back my results.

[burtyb]

@JamesH65 even if it can't "break in to 'something'" there are costs for the provider to visit/use remote hands to revert say a malicious EEPROM update that prevents network booting which could be easily prevented if hardware write protection was an option.

[JamesH65]

This is a niche issue on Model B and not supported.

Given current trends, "desktop replacement" is likely a much smaller niche than cloud. Upon noting Google's interest in it, being able to write protect the firmware is a big selling point (not only for the compute module) and should, in my opinion, be mentioned in the feature summary as well as documentation.

Absolutely disagree. Sticking a load of Pi's in the cloud (Yes, its already done) is nowhere near desktop sales. Might change, but not in the near future.

The original documentation said "However, it is possible to physically write-protect both EEPROMs via a simple resistor change on the board. Details will be published in the schematics." The use of the word simple made me think this would be simple rather than not supported. That original document would have been quite easy to understand if details of how to write protect the firmware had actually been provided in the schematics as claimed.

Moving forward, it would be great if a more technical document were produced that describes how to write protect the firmware and firmware settings. I would be happy to look that over, try the procedure and report back my results.

Apparently, connecting TP5 to earth will write protect the EEPROMS (VLI and boot) on the Pi4. I just found this out, would have been useful to know at the start of this thread.

[stiell]

I've managed to lock both EEPROMs, I'll try to get docs and code out soon.

[stiell]

Apparently, connecting TP5 to earth will write protect the EEPROMS (VLI and boot) on the Pi4. I just found this out, would have been useful to know at the start of this thread.

Note that, as I mentioned in my earlier comment, the hardware modification alone is not enough to enable write protection. The chips also need to have write protection bits enabled by software.

[cleverca22]

While the tenant did not have physical access, the person cleaning the Pi between tenants does not readily have physical access either. It simply does not scale well to go to the data center, pass through security and insert the firmware reset SD card into the corresponding Pi between each tenant. Fortunately, such efforts would not be needed if after the network boot option was set, it couldn't be changed.

@ejolson2005 using OTP66, you can configure a gpio pin to disable recover.bin loading from the SD card you could then leave that factory-reset card in the pi at all times, and use GPIO to enable it remotely at your choosing

but you still need to protect the SD card itself, so write-protecting the eeprom's is probably a saner idea

there are also pure software ways to brick a pi4, but i wont say what for obvious reasons

edit: how did i un-assign somebody by accident? i didnt click anything in that part of the ui, and i shouldnt even have access??

[timg236]

Since write protect this is required for CM4 we are looking at making that change available earlier since shorting TP5 is relatively low risk. The default settings and compatibility with older versions of recovery.bin for VL805 write-protect aren't settled yet though.

[meeDamian]

but you still need to protect the SD card itself

That's actually not as hard to do as it would seem (see: https://github.com/BertoldVdb/sdtool), albeit it does come with some caveats:

1. Write protection is permanent, but given the price and ubiquity of SD cards, that might not be a big issue,
2. Most software doesn't really understand what's going on, and will incorrectly think the write succeeded, but that also might not be a huge issue given how the SD card is supposed to only store recovery OS.

[timg236]

I've uploaded a test version of recovery.bin plus bootloader which can optionally set the bootloader and VLI EEPROMs as write-protected, which can be secured by grounding TP5.

Please read the release notes fully before using this.

Older versions of the bootloader and vl805 tools do NOT automatically clear the WP bits on the VL805 EEPROM so you have to use this test version (which includes fixed vl805 user space tool) to clear the write protect bits.

I think most people who want this behaviour would use an sd-card to toggle write protect (see release notes). It should work with self-update mode but that's more confusing to use and only briefly tested so far.

https://github.com/raspberrypi/rpi-eeprom/tree/eeprom_write_protect https://github.com/raspberrypi/rpi-eeprom/blob/eeprom_write_protect/firmware/release-notes.md

Updated recovery.bin and pieeprom-2020-07-14.bin

Once we are happy that it's safe enough this will become a normal rpi-eeprom-beta release.

Please submit bugs as rpi-eeprom issues and be as specific as possible about version numbers, UART logs are probably essential here.

[ln2max]

Apparently, connecting TP5 to earth will write protect the EEPROMS (VLI and boot) on the Pi4. I just found this out, would have been useful to know at the start of this thread.

Note that, as I mentioned in my earlier comment, the hardware modification alone is not enough to enable write protection. The chips also need to have write protection bits enabled by software.

I currently have an issue open in Flashrom to add this support: https://github.com/flashrom/flashrom/issues/185

[JamesH65]

closing as the original issue, write protections, has now been in the system for some time. The Flashrom stuff is unrelated.