Applying external voltage to X230's flash chips is considered dangerous

[i-c-o-n]

We had very few reports of permanent bricks @ coreboot over the years. But all those I can remember were about the ThinkPad X230 and involved VCC supplied directly to the flash chip (which you should never do anyway unless the VCC is protected with a diode or something because you'll violate the board's power sequencing).

Therefore, we recommend to use the laptop's own PSU to power the flash chip. On the X230, the flash chips should be powered when the laptop is in S5, listening for Wake-on-LAN messages. Usually you can enter that state when you remove all batteries, plug a LAN cable (with active remote) and then plug the PSU.

Although I've never heard of permanent bricks using that method, the actual problem is not well studied. So I'd recommend to flash only the minimum externally, i.e. the 4MiB chip and optionally unlock the firmware descriptor.

NB. saying the BIOS only fills the second chip is not accurate (it starts at 5MiB in the first chip), though that simplification works in the coreboot case with CBFS_SIZE<=4MiB.

[Th3Fanbus]

Any comments on this?

[merge]

have you had problems with our documented procedure? If so, let's change the documentation. If it's doene in a clean way, you could at least add a chapter about WoL powering. but i'd rather only have that if really necessary.

I agree that it makes sense to not flash externally often. We probably should change that part of the docs where we say "flashing internally is dangerous".

[pgera]

It is possible to avoid external flashing altogether if you just want to go from the lenovo bios to coreboot. This won't help you write me, gbe and fd regions without an external programmer, but you can replace the bios region internally. For most users, that would be far more convenient. You should have an external programmer though as a contingency. I don't have an x230, but the general principle would still apply. I can provide some instructions if there is interest.

[Th3Fanbus]

How is that possible, if there are set protected ranges that prevent flashrom using the internal programmer from writing to the needed flash chip region? Do note this has nothing to do with the IFD regions, the protected ranges are set (or not) independently of the IFD settings.

[pgera]

I did this for w530. It involves rolling back the bios to an older version and using an smm vulnerability to bypass spi write protection.

# Tested on W530 with BIOS g5uj21us. 
# You can rollback to older BIOS with winflash64.exe /sd /file <path_to_.fl_file> after disabling rollback protection in the bios settings

# BIOS_CNTL
sudo chipsec_main -m tools.uefi.s3script_modify -a add_op,pci_wr,0x1f00dc,0x9,1

# FLOCKDN
sudo chipsec_main -m tools.uefi.s3script_modify -a replace_op,mmio_wr,0xFED1F804,0x6008,0x2

# S3 suspend after this. Can be done with systemctl suspend

# After resume:
sudo chipsec_util mmio write SPIBAR 0x74 0x4 0xAAF0800
sudo chipsec_util mmio write SPIBAR 0x78 0x4 0xADE0AD0
sudo chipsec_util mmio write SPIBAR 0x7C 0x4 0xB100B10
sudo chipsec_util mmio write SPIBAR 0x80 0x4 0xBFF0B40

# Test with this:
sudo chipsec_main -m common.bios_wp
# Expected output:
[x][ =======================================================================
[x][ Module: BIOS Region Write Protection
[x][ =======================================================================
[*] BC = 0x09 << BIOS Control (b:d.f 00:31.0 + 0xDC)
    [00] BIOSWE           = 1 << BIOS Write Enable 
    [01] BLE              = 0 << BIOS Lock Enable 
    [02] SRC              = 2 << SPI Read Configuration 
    [04] TSS              = 0 << Top Swap Status 
    [05] SMM_BWP          = 0 << SMM BIOS Write Protection 
[-] BIOS region write protection is disabled!

[*] BIOS Region: Base = 0x00500000, Limit = 0x00BFFFFF
SPI Protected Ranges
------------------------------------------------------------
PRx (offset) | Value    | Base     | Limit    | WP? | RP?
------------------------------------------------------------
PR0 (74)     | 0AAF0800 | 00800000 | 00AAF000 | 0   | 0 
PR1 (78)     | 0ADE0AD0 | 00AD0000 | 00ADE000 | 0   | 0 
PR2 (7C)     | 0B100B10 | 00B10000 | 00B10000 | 0   | 0 
PR3 (80)     | 0BFF0B40 | 00B40000 | 00BFF000 | 0   | 0 
PR4 (84)     | 00000000 | 00000000 | 00000000 | 0   | 0 

[!] None of the SPI protected ranges write-protect BIOS region

Edit: While you can't write to the ME region without an external programmer, you can put it in a soft temporary disable state. OEM BIOS has that option in advanced settings. So it should be possible from coreboot too. Running metool would give the following output:

MEI found: [8086:1e3a] 7 Series/C216 Chipset Family MEI Controller #1

ME Status   : 0x1e030185
ME Status 2 : 0x12280106

ME: FW Partition Table      : OK
ME: Bringup Loader Failure  : NO
ME: Firmware Init Complete  : NO
ME: Manufacturing Mode      : NO
ME: Boot Options Present    : NO
ME: Update In Progress      : NO
ME: Current Working State   : Normal
ME: Current Operation State : Bring up
ME: Current Operation Mode  : Soft Temporary Disable
ME: Error Code              : No Error
ME: Progress Phase          : BUP Phase
ME: Power Management Event  : Clean global reset
ME: Progress Phase State    : ME in temp disable

ME: Extend SHA-256: [long hash here]

ME: failed to become ready
ME: failed to become ready
ME: GET FW VERSION message failed

[swiftgeek]

As long as you wire everything correctly and use proper voltage levels¹ and don't inflict ESD damage to the board, and limit current going to the board² and unit has »diode« installed (like D98 on regular x230 - Dasher-2 ) it is a safe thing to do if you know what you are doing. Attaching testclip to 25xx SPI flash is not as easy as plugging in USB-C :)

But powering via AC adapter is definitely more dangerous than externally with current limiting, especially when diode is present, where expected current usage will be so low my multimeter won't register it and only flash chip(s) are powered - VCC3LAN_SPI node.

And don't get me started on how many people attempted to flash mosfet because it's soic8 too...

¹ Example of device that uses 5V levels are many of the CH341A adapters found on aliexpress. I would strongly advise against using them unmodified. Example modification required for 3v3 levels ² Setting current limit is not providing perfect protection from connecting testclip wrong way around but it will significantly decrease likelihood of damage when those human errors happen

[merge]

@swiftgeek yes, thanks for this summary :) we could add some of this to our docs.

@pgera if we can 1. downgrade, 2. read all, 3. run ifdtool and write that back, and have at least a plan how to automate that process a bit, we should test this. thanks for bringing this up. Do you know what vulnerability we should look for in Lenovo's bios changelogs; or how old the version is you use there?

[pgera]

I think it's this one https://support.lenovo.com/us/en/solutions/len-8324, but could have been something else too since there was more than one issue from what I remember. You can see in the x230 bios changelog that this was patched in 2.67. For my experiments, I used 2.58 for the w530.

[pgera]

Also, you may have to make some minor tweaks to the chipsec commands above if there are any differences between x230 and w530. There shouldn't be, but chipsec is generally quite useful in determining addresses and ranges if you need to change anything. And if you have trouble with ifdtool or flashrom, try intel's fptw64. I haven't tried writing anything internally with coreboot's utils, but fptw works from experience.

[eloydegen]

@pgera Are you aware of any other vulnerabilities to make the same procedure possible on systems running a classic BIOS instead of UEFI, that's also doable with CHIPSEC? Would be cool to make it possible to run coreboot on more machines without external flashing involved (although that is risky if people don't have the backup equipment)

[pgera]

@eloydegen , sorry not too familiar with the general landscape for other systems.

[camoz]

I would advise against rolling back the BIOS version to enable software-based flashing of coreboot. The BIOS update also updates the ME and EC firmware and sometimes has security fixes for them. Even when stripping the ME down (aka "disabling the ME"), some modules still remain, and IIRC there were CVEs in the past which affected those remaining modules. For the EC firmware, see e.g. CVE-2019-6171. EC updates also contain bug fixes or functional improvements like for battery charging and so on.

[cuvtixo]

Usually I refrain from contributing, sorry if this is irregular. CVE-2019-6171 was rather extraordinarly late in the life of this laptop, and the description itself said it effected "BIOS versions of older ThinkPad systems". The last firmware update was both late and fairly minor. Are you suggesting that there might be more in the future? If there were functional improvements for battery charging by Lenovo for the X230 now, it would be rather shocking. (lol, sorry). Your advice, camoz, would make some sense on current "production" hardware, but on an 8 year old convertible-tablet-laptop? (I don't think that class of machinery even is made anymore) I think you have lost perspective on this. Improvements on battery charging and whatever else are really of secondary considerations for X230at this stage, a machine that's as likely as not to be used specifically for experimenting upon, and making modifications, such as an IPS screen, updating the keyboard, etc. Librem and System 76 are putting out current laptops, so the need for holding on to these aging X230s simply for Heads and Skulls has been much lowered.

[camoz]

I only wanted to point out that rolling back the BIOS might have unwanted side effects. Since this issue is still open and the discussion above was not about a single topic, and also improving docs, I thought putting that here doesn't hurt. Also, the X230 might be an old laptop, but if some people are still using it it seems reasonable to me to go with the latest ME/EC firmware.

[cuvtixo]

It's a little cheeky of me, admittedly. I gave one to my elderly parents and I've modified my own a fair bit, and I now have a Librem. Therefore my view on it. I think you're comment would be very wise in the context of another machine.it

[jcholsap]

The way I remember it: The CVE referenced pertains to people being able to flash the BIOS firmware, i.e., skulls and hamish. The fix from Lenovo was a firmware update that included saving a hash of the BIOS inside the MEC SDRAM. In short, if you update your firmware per the CVE, you won't be able to mod the BIOS or EC. If you updated and wanted to rollback, then you'd need to JTAG the MEC.

Stripping the BIOS down to just the bare essentials and "disabling" the EC is the most secure path. Even as recently as November, CVE-2020-8752 rated critical, 9.2/10, results from the Intel AMT "feature" enabled in the BIOS.

I'd like to point out that everyone here is a hobbyist, not a firmware engineer, and learning from the experience is part of the fun.

[vkhodygo]

I know my questions are not related directly to this issue, but it looks like the best place to ask.

I'm trying to port coreboot to one of my old machines. The problem is the voltage on pin 8 is zero even when the charger is plugged in. I have to actually turn it on to get the voltage to 3v. Unfortunately, I failed to find any similar machines/issues.

After some extensive search I've found the schematic, and it looks like there is no diode between VCC and EC meaning all that current from the programmer goes to /dev/null.

Any suggestions regarding how to fix this without frying the board/(extra) soldering?

[i-c-o-n]

I know my questions are not related directly to this issue, but it looks like the best place to ask.

You can try to contact flashrom folks. The IRC channel (also joinable via Matrix) works best for such issues.

Any suggestions regarding how to fix this without frying the board/(extra) soldering?

If the flash chip has a /HOLD pin and the board a weak pull-up resistor connected to it, the best option may be to boot from an external flash chip (or emulator) and then flash the onboard chip internally.

[vkhodygo]

@i-c-o-n Thanks for the info, I'll try that.

flash the onboard chip internally

The problem is I have image to flash in the first place, I need to read from the chip first.