search

linuxboot / heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible boot environment for laptops, workstations and servers.
https://osresearch.net/
GNU General Public License v2.0
1.4k stars 180 forks source link

Provide IFD, Gbe and ME blobs #307

Closed osresearch closed 3 years ago

osresearch commented 6 years ago

The 12 MB x230 ROM image isn't flashable since it doesn't include the IFD and ME sections. This isn't normally a large problem for updates since flashrom-x230 won't touch the ME region, but it does lead to problems for initial install.

tlaurion hijacking top post here to give actual status in a glimpse.

Mitigations from dropping those binaries would be to:

So the result of the ongoing #703 PR is a PoC of that, which requires Heads to fake MAC address on boot to not have a MAC collision on the same LAN network segment if many Heads xx30 empowered machines are present.

Todo:

merge commented 6 years ago

This and #375 is really only one issue IMO :)

tlaurion commented 6 years ago

I think it might a false problem.

For both x220 and x230 and other 2 spi chips boards, the IFD should be provided inside of Heads (./initrd/etc/x230-layout.txt), accompanied with a flash script (/initrd/bin/flashrom-x230.sh) for Heads to be able to flash internally, the IFD containing the ME region to not touch, though that region could be reduced a lot, no?

@osresearch : would flashtool be able to deal with reading the IFD in flash dynamically and determine the real used ME region? I mean. On the x230, there is plenty of space when ME modules are trimmed and the free space is concatenated:

./me_cleaner.py ~/Documents/Firmwares/ME/x230/MeRelocated.rom -S  -O x230_me_cleaned.rom -r -t --extract-me extracted_me.rom
Full image detected
The ME/TXE region goes from 0x3000 to 0x500000
Found FPT header at 0x3010
Found 1 partition(s)
Found FTPR header: FTPR partition spans from 0xd00 to 0xcad00
ME/TXE firmware version 8.1.40.1416
Public key match: Intel ME, firmware versions 7.x.x.x, 8.x.x.x
The AltMeDisable bit is NOT SET
Reading partitions list...
 FTPR (0x00000d00 - 0x0000cad00, 0x000ca000 total bytes): NOT removed
Removing partition entries in FPT...
Removing EFFS presence flag...
Correcting checksum (0x2f)...
Reading FTPR modules list...
 UPDATE           (LZMA   , 0x04d207 - 0x04d3c5       ): removed
 ROMP             (Huffman, fragmented data, ~2 KiB   ): NOT removed, essential
 BUP              (Huffman, fragmented data, ~56 KiB  ): NOT removed, essential
 KERNEL           (Huffman, fragmented data, ~135 KiB ): removed
 POLICY           (Huffman, fragmented data, ~91 KiB  ): removed
 HOSTCOMM         (LZMA   , 0x04d3c5 - 0x05419f       ): removed
 RSA              (LZMA   , 0x05419f - 0x0593f5       ): removed
 CLS              (LZMA   , 0x0593f5 - 0x05eb8a       ): removed
 TDT              (LZMA   , 0x05eb8a - 0x065280       ): removed
 FTCS             (Huffman, fragmented data, ~18 KiB  ): removed
 ClsPriv          (LZMA   , 0x065280 - 0x065661       ): removed
 SESSMGR          (LZMA   , 0x065661 - 0x073f8b       ): removed
Relocating FTPR from 0xd00 - 0xcad00 to 0xd00 - 0xcad00...
 Adjusting FPT entry...
 Adjusting LUT start offset...
 Adjusting Huffman start offset...
 Adjusting chunks offsets...
 Moving data...
The ME minimum size should be 98304 bytes (0x18000 bytes)
The ME region can be reduced up to:
 00003000:0001afff me
Setting the AltMeDisable bit in PCHSTRP10 to disable Intel ME...
Extracting and truncating the ME image to "extracted_me.rom"...
Checking the FTPR RSA signature of the extracted ME image... VALID
Checking the FTPR RSA signature... VALID
Done! Good luck!
tlaurion commented 6 years ago

In this case, following the previous output from me_cleaner:

The ME region can be reduced up to:
 00003000:0001afff me

/etc/x230-layout.txt ME region could be modified from 

00000000:00000FFF Descriptor     
00001000:00002FFF GBe                   
00003000:004FFFFF ME              
00500000:00BFFFFF BIOS

to: 00003000:0001AFFF ME

So we could use the output of me_cleaner to adapt the /etc/x230-layout.txt

Is there something i'm missing?

merge commented 6 years ago

no, I think you're right. The question is mostly whether we need the space and want to become dependent on me_cleaner being applied.

Am 20. April 2018 22:41:23 MESZ schrieb tlaurion notifications@github.com:

In this case:

00000000:00000FFF Descriptor     
00001000:00002FFF GBe                   
00003000:004FFFFF ME              
00500000:00BFFFFF BIOS

ME could be reduced to: 00003000:0001AFFF ME

Is there something i'm missing?

-- Martin Kepplinger http://martinkepplinger.com sent from mobile

merge commented 6 years ago

@osresearch yes the 12m isn't flashable which is dangerous when flashing externally (first-time install) and people get it wrong (and the head wiki might even be misleading a little).

FWIW, specifically (only) for the X230, there's now a tiny project named Skulls that basically tries to hide as much as possible and simplify external flashing / unlocking the IFD (and applying me_cleaner). There is a script to install Heads (from Linux) too.

merge commented 5 years ago

this comment actually belonges in here more: https://github.com/osresearch/heads/pull/424#issuecomment-441074700

tlaurion commented 5 years ago

@osresearch @merge @flammit : The more I think about it, the more I start to believe that the same approach as Purism should be taken to download and neutralize ME binary blob before inserting it into Heads's 12MB rom, with a valid IFD being generated for that ROM.

The x230 flash.sh specifics could then overwrite the whole SPI instead of just the bios region.

What do you think? Modifying the scripts.

tlaurion commented 5 years ago

@merge @flammit @osresearch

Missing some pieces of the puzzle.

merge commented 5 years ago

why? what's the benefit when flashing internally on the system?

tlaurion commented 5 years ago

@merge more space, the possibility to update ME internally following probable changes in the ROMP and BUP modules and to have a valid IFD instead of only being able to flash the bios region since the layout file is not provided anymore. Flashrom 1.0 upgrade and related flash.sh modifications wiped it.

But basically, more usable space in SPI flash. Maybe a false need, again.

tlaurion commented 5 years ago

@merge said in #424 which is now merge:

We need a plan here :) we don't even know what to do yet

    First decision to make: we should agree that generating the 4M and 8M images of the x230 board is wrong in any case. We'll never be able to safely generate a flashable thing of 12M. And there's no point in having the splits other than flashing externally, which is always wrong.
    Also, flashing externally is only done the first time. And that's what the x230-flash board is for, which makes the x230 4M/8M split even more misleading. But then, x230-flash is not sufficient. At least touching the other chip's IFD is necessary.

    So, second decision to make: Do we want to support "bootstrapping" explicitely inside of Heads?
        If so, let's add scripts that unlock the IFD on the one chip and flash x230-flash on the other, and guide users through all of it.
        If not, drop x230-flash completely and simply add "unlocked IFD" to the requirements and add a script or (even better?) live-USB-image that flashes x230 from outside of Heads, which makes "bootstrapping" quite neat.

thoughts?

I think the x230-flash should deal with it internally. If i get things right from skulls, it means that x230-flash misses ifd tool to unlock the IFD from within. @merge?

merge commented 5 years ago

ifdtool's "unlock" has to be applied to the 8M chip's region. x230-flash (just like a skulls image) uses 4M (the 4M chip) and thus makes it possible to hardware-flash "read, unlock, write" to the 8M chip, which makes the process as easy as possible (as of now). That documentation is what's important.

x230-flash or x230 shouldn't include ifdtool in initrd. For future write-protection solutions we won't use IFD. We just need it to be unlocked once.

tlaurion commented 5 years ago

Interesting points posted in #564

@merge

I haven't really looked into the current state of the x230-flash "board". IMO it shouldn't need gpg at all. It's for bootstrapping only, right?

gpg is not there anymore, since GPG2 is too big and x230-flash is used only to bootsrap. @shamen123 : the 12mb, internally flashed version is used for everything else then initial flashing of Heads.

Once the 4M "board" itself is in a nice shape (flash-gui, ... can be tested independently of the bootstrapping problem), we can think about how to help users with IFD unlocking. I can imagine a similar solution to https://github.com/merge/skulls/blob/master/x230/external_install_bottom.sh and good documentation. That script can directly be used actually... If it wasn't for "x230-flash" (a flash-only solution independent of the hard disk), we could even say "install Skulls first, and then come back here and flash Heads for your running Linux distro", but it's of course nicer to have a flash-only solution.

Unfortunately, following #517 and 4mb limit, having a flash-gui displayed in FBWhiptail or Whiptail only, won't fit in 4Mb.

I will look again on the Skull's external_install_bottom.sh script, since I can't still wrap my head about how we could deal with this one shot from the external, flashing system. Never actually played with Skull. Will try to find some time.

@shamen123

i did make write up notes at the time and they are lurking around on one of my machines somewhere, which may be of use to others. but that was based on 0.2.1 generic init, so its likely already out of date.

The process hasn't change a lot since that time. Chip that script in?

merge commented 5 years ago

Interesting points posted in #564

@merge

I haven't really looked into the current state of the x230-flash "board". IMO it shouldn't need gpg at all. It's for bootstrapping only, right?

gpg is not there anymore, since GPG2 is too big and x230-flash is used only to bootsrap. @shamen123 : the 12mb, internally flashed version is used for everything else then initial flashing of Heads.

Once the 4M "board" itself is in a nice shape (flash-gui, ... can be tested independently of the bootstrapping problem), we can think about how to help users with IFD unlocking. I can imagine a similar solution to https://github.com/merge/skulls/blob/master/x230/external_install_bottom.sh and good documentation. That script can directly be used actually... If it wasn't for "x230-flash" (a flash-only solution independent of the hard disk), we could even say "install Skulls first, and then come back here and flash Heads for your running Linux distro", but it's of course nicer to have a flash-only solution.

Unfortunately, following #517 and 4mb limit, having a flash-gui displayed in FBWhiptail or Whiptail only, won't fit in 4Mb.

not yet at least :) but alright. There's no need to have a GUI for that one job.

I will look again on the Skull's external_install_bottom.sh script, since I can't still wrap my head about how we could deal with this one shot from the external, flashing system. Never actually played with Skull. Will try to find some time.

It's nothing special. Just a different 4M coreboot build, and thus it has all the bootstrapping-benefits that we need equally need to take advantage of for "x230-flash" here. Basically, you can even directly replace a pre-built "skulls" image with an "x230-flash" image and directly use skulls' "external" scripts - one per flash chip.

@shamen123

i did make write up notes at the time and they are lurking around on one of my machines somewhere, which may be of use to others. but that was based on 0.2.1 generic init, so its likely already out of date.

The process hasn't change a lot since that time. Chip that script in?

tlaurion commented 4 years ago

605

tlaurion commented 4 years ago

@merge : There is not enough space anymore in the BIOS region (coreboot cbfs defined space) as it is to do anything else.

Playing around and applying this, including ifd and me in rom (while modifying flash.sh for it to reflash whole 12mb, not just BIOS region), there is now around 5mb flash that just appeared magically.

Looking at how Purism distributes its binaries through whole rom download for their board, there doesn't seem to be any problem redistributing ifd and me, which are basically included in their downloadable image, and extracted by their scripts.

@kylerankin ? @zaolin : Would there be a legal problem redistributing me.bin (98kb shrinked bin) and resulting ifd.bin directly from a x230 blobs directory on the repo?

What I propose:

This way, reading roms would only be useful to have backups of them, but there would be no real reason to even run me_cleaner anymore. Just download GitlabCI artifact, and flash both chips.

x230 me_cleaner command generating blobs would ressemble this: python me_cleaner.py -S -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_down_dump.bin This neuter+deactivate(AltMEDeactivateBit) Intel ME leaving only BUP and ROMP, trim resulting space, remove IME access to other flash regions, and export both ifd.bin and me.bin that would be distributed and included in expended CBFS BIOS region under x230 coreboot config file.

What do you think? Playing around to include thin-provisioning-tools right now to measure OEM distributed image directly from distributed /boot with dm-verity at reownership. More space is needed and my quest to reduce firmware size has not gone very far.

@marmarek: That would be useful too if fwupd subproject goes farther in providing firmware upgrades from QubesOS over sys-whonix from dom0 (so not a Heads task, but QubesOS task.) Heads could look into directory for signed firmware file, validate against distributing key inside of rom, and compute new measurements based on firmware parts to implement forward sealing of firmware upgrades.. Any thoughts on that are also welcome.

Playing with this here

osresearch commented 4 years ago

The ifd.bin should be included; I don't think there are any licensing issues. me.bin I'm not sure about.

Intel has a very reasonable redistribution clause for their microcode updates that came about from the Debian pushback as well as for the FSP.

However, the ME team has not had any similar sort of large scale push them to change their license. Lots of us have asked, but I've never received a response.

Fetching it from a separate blobs repo would at least avoid tainting the main Heads tree, which seems to be the coreboot approach. This also avoids bloating up the tree with how ever many devices end up being supported.

merge commented 4 years ago

@merge : There is not enough space anymore in the BIOS region (coreboot cbfs defined space) as it is to do anything else.

Playing around and applying this, including ifd and me in rom (while modifying flash.sh for it to reflash whole 12mb, not just BIOS region), there is now around 5mb flash that just appeared magically.

I guess we should just do that. Might be the easiest way to keep the x230 supported. I might think that through sometime... (we need more space for vboot later anyways).

(not now but for the future, I'd be interested: would we then need a new vboot fmd file in coreboot too? how would that look like?)

merge commented 4 years ago

I tested this too and it works nicely: https://github.com/merge/heads/commit/4cbab95691d169aa4d7e52f27717be5a5a7fddcb 5M of free space. we need to be careful how to make the transition:

We change the IFD, so flashing from inside of Heads won't (conveniently) work! I guess before we do a Heads-release, we can and should do this and force flashing from outside of Heads (ifd, me and bios regions (we can write a script) from userspace or 4M/8M from external hardware).

do you see any way we could avoid that "break" for users? IMO the state of the project would allow us to do it and say "please flash from outside of Heads, once now", when doing a release?

tlaurion commented 4 years ago

590 can be closed when this is resolved.

tlaurion commented 4 years ago

@osresearch @merge @kylerankin @MrChromebox : In the goal of deploying descriptor.bin and me.bin (and other blobs), I was wondering what would be the best method to do this globally for all boards.

[user@heads-x230-libremkey heads]$ find build/coreboot-4.8.1/3rdparty/ -name me.bin -o -name descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/stout/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/stout/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/parrot/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/parrot/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/link/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/link/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/butterfly/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/google/butterfly/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/samsung/stumpy/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/samsung/stumpy/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/samsung/lumpy/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/samsung/lumpy/me.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/intel/emeraldlake2/descriptor.bin
build/coreboot-4.8.1/3rdparty/blobs/mainboard/intel/emeraldlake2/me.bin
merge commented 4 years ago

@osresearch @merge @kylerankin : In the goal of deploying descriptor.bin and me.bin (and other blobs), I was wondering what would be the best method to do this globally for all boards.

* Put a download script under blobs/x230 that fetches the binaries and store them locally in that directory (current way).

* Patch coreboot-blobs with additional files (see below)?

* Download github archive from another repo for a specific commit ID and extract into `mainboard/lenovo/x230/`?

go with the first, current way because that's straight forward and only simple additions. Moving the extract/download scripts out of the way in favour of putting stuff in coreboot's blobs repo is a different task that would make sense for all boards, but doesn't have to block this problem: Create a patch to the coreboot blobs repo, put it up for review and in the meantime, include it as a patch in Heads that includes all the binaries.

tlaurion commented 4 years ago

@merge Best way for newer coreboot versions seems to be to have them included directly into coreboot-blobs, IMOHO, not breaking any licence, as done previously (see precedent post).

Maybe have me_cleaner as a dependency in boards, and apply me_cleaner specific board relative command directly on them? python me_cleaner.py -S -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_down_dump.bin

Heads could then produce externally flashable legit images (4mb and 8mb images), directly from the board configuration, that would not require to flash only the BIOS ifd region in flash script, removing the need for x230-flash board (and other coming boards that would need to follow the same path if not properly fixed right now), permitting LVFS hosted images and fwupd port to QubesOS to do a proper job without being hackish and just drop the firmware image into /boot to be proposed to be flashed if signature matches. And that would also easily resolve CI problems in doing things in a more complicated, manual and hackish way.

Note that it might be a good time to split board configurations into specialized ones, since that space is not required as of right now by all the boards. I personally am fond of this idea, where board's flash specific commands are brought back into board configurations . That idea would address @merge's concern:

do you see any way we could avoid that "break" for users? IMO the state of the project would allow us to do it and say "please flash from outside of Heads, once now", when doing a release?

Getting away of x230 generic board would require a first, initial external flashing for specialized board configs, requiring the additional space for additional tools (ex: x230-hotp, x230-thin-verified, or more simply: x230-NeuteredME). Note that boards passing to newer coreboot version will need more space to have VBOOT+measured boot. Now may be a good time to have a maintainable solution, where initial external flashing will be required one way or the other if IFD changed or if ME partitions are still locked and non-internally flashable.

@kylerankin @merge @flammit @osresearch : Thoughts welcome.

tlaurion commented 4 years ago

@BrianRichardsonIntel: any restrictions on redistributing neutered ME blobs and how to properly go forward with this?

Is the path to currently host those files remotely in the form of ROMs where they are already neutered and having to extract IFD and ME manually prior to build coreboot against them to build clean roms?

Or could we just transparently distribute those neutered ME and expended IFD blobs directly in the blobs directory of Heads, with a version from which it came from and corresponding hashes (and licensing?)

Any guidelines on https://github.com/osresearch/heads/issues/307#issuecomment-552961256 would be greatly appreciated.

tlaurion commented 4 years ago

@merge

@osresearch @merge @kylerankin : In the goal of deploying descriptor.bin and me.bin (and other blobs), I was wondering what would be the best method to do this globally for all boards.

* Put a download script under blobs/x230 that fetches the binaries and store them locally in that directory (current way).

* Patch coreboot-blobs with additional files (see below)?

* Download github archive from another repo for a specific commit ID and extract into `mainboard/lenovo/x230/`?

go with the first, current way because that's straight forward and only simple additions. Moving the extract/download scripts out of the way in favour of putting stuff in coreboot's blobs repo is a different task that would make sense for all boards, but doesn't have to block this problem: Create a patch to the coreboot blobs repo, put it up for review and in the meantime, include it as a patch in Heads that includes all the binaries.

@MrChromebox @flammit @osresearch : The problem with this approach is that no Continuous Integration platform will produce those boards reproducible artifacts for users. Another approach is needed, respecting Intel status quo of silence/challenging it, while still providing required space and neutering where possible where they won't.

I still don't get it personally. OEM distribute it. You can extract neutered from SPI but can't put it online? This silence seems to be a way to not prohibit it nor promote it. Most legal people I've talked to said Intel would never do anything about it. I've contacted them 3 times by different ways to come to an agreement, without any answer back. And i'm not the only one, everyone interested in the topic says the same.

I may be rogue in my ways a bit, but personally, I would simply put the neutered blobs online and wait for Intel to ask to remove them, which would start talking about the issue for real. Any thoughts welcome, once again.

MrChromebox commented 4 years ago

I may be rogue in my ways a bit, but personally, I would simply put the neutered blobs online and wait for Intel to ask to remove them, which would start talking about the issue for real

100% agree

tlaurion commented 4 years ago

@MrChromebox, @osresearch @merge @flammit @ : so let's do it? That would simplify the whole process for adventurous users and OEM from CI builds directly producing artifact, instantaneously.

merge commented 4 years ago

I may be rogue in my ways a bit, but personally, I would simply put the neutered blobs online and wait for Intel to ask to remove them, which would start talking about the issue for real

100% agree

I also agree here.

tlaurion commented 4 years ago

@osresearch : three cabals are in. FSFE legal dpt contacted in the scope of Accessible Security Grant. Ready to take some risks to challenge status quo?

tlaurion commented 4 years ago

@osresearch @merge @kylerankin @MrChromebox @SebastianMcMillan : The same issue is repeating itself over and over again for all to be supported boards having ME if we want to have other tools included in Heads and CI reproducibility, latest being #690.

We have to agree on a plan. Go!

snmcmillan commented 4 years ago

I may be rogue in my ways a bit, but personally, I would simply put the neutered blobs online and wait for Intel to ask to remove them, which would start talking about the issue for real

100% agree

I also agree here.

Count me in as well.

tlaurion commented 4 years ago

Coreboot refuses to add blobs in coreboot-blobs without license, it seems

tlaurion commented 4 years ago

@merge @MrChromebox @SebastianMcMillan @osresearch

The more I think about it, the more I think we have to seperate boards from now on.

693 , for example, could be x220-libremkey t420-libremkey, and should point to proper neutering instructions placing well carved ifd and me blobs to build coreboot for a specific Lenovo ROM update.

Maybe the way to resolve #307 is to actually borrow/rewrite code from @n4ru 's 1vyrain to extract ME from original Lenovo BIOS upgrades directly, and patch Heads Makefile/coreboot-blobs script in so that when CONFIG_NEUTERED_ME is explicitely stated in board config, board blob scripts are called to download those original roms, apply me_cleaner the way we need it to be called and drop unlocked expended ifd.bin and neutered+deactivated me.bin and other needed boards blobs locally to be able to build Heads from CI instances and have reproducible builds....

eganonoa commented 4 years ago

@tlaurion

The more I think about it, the more I think we have to seperate boards from now on.

693 , for example, could be x220-libremkey t420-libremkey, and should point to proper neutering instructions placing well carved ifd and me blobs to build coreboot for a specific Lenovo ROM update.

As someone who struggled initially to figure out how to get the Nitrokey to work with Heads (in part confused by the different naming conventions: HOTP/LibremKey/Nitrokey), I think having separate boards for Heads with a key and Heads without a key (for all systems) would make a ton of sense. That would avoid people having to mess around with the various .config files and would just allow them to make BOARD=x220 or _make BOARD=x220HOTP (or something like that). It would also allow for a simple addition to README.md and would help you figure out who can and who cannot do the testing of how well the keys work on different systems. Potentially it would also be useful for Purism and Nitrokey to point to the right way to do this in their own manuals for the Librem Key and the various compatible Nitrokeys?

tlaurion commented 4 years ago

Searching ways to actually get the rom images directly from Lenovo and go from there to do a PoC.

Adventure in the goal of putting those blobs in total legaility in a CI:

Question: is there any linux tools that can decompress FL2 content to rom images as instructed (non working) from coreboot deprecated guide here?

tlaurion commented 4 years ago

Most interesting project that could do the job so far seems to be uefi-firmware-parser, which supports FL2 with superbruteforce (--superbrute) statement, but extracted regions are not fitting the bill.

me_cleaner doesn't pick up on any complete 8MB file nor ME region, on which it could work on: wget https://download.lenovo.com/pccbbs/mobiles/g2uj31us.exe && innoextract ~/g2uj31us.exe && echo -e "\n\n EXTRACTING FL1\n\n\n" && /usr/bin/python2.7 /usr/local/bin/uefi-firmware-parser ./app/G2ETB5WW/\$01D3000.FL1 -e && echo -e "\n\n\n EXTRACTING FL2 \n\n\n" && /usr/bin/python2.7 /usr/local/bin/uefi-firmware-parser ./app/G2ETB5WW/\$01D3000.FL2 -e --superbrute && echo -e "\n\n\n Testing me_cleaner against all extracted files.... output given only on successful files... \n\n\n" && find ./pfheader* ./capsule-Capsule/ ./volume-0/ -type f | while read filename; do python ~/me_cleaner/me_cleaner.py -S -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin $filename &>/dev/null; if [ $? -eq 0 ]; then echo $filename; fi; done;

Nor match any file that would match ME envelope (more generic me_cleaner command): wget https://download.lenovo.com/pccbbs/mobiles/g2uj31us.exe && innoextract ~/g2uj31us.exe && echo -e "\n\n EXTRACTING FL1\n\n\n" && /usr/bin/python2.7 /usr/local/bin/uefi-firmware-parser ./app/G2ETB5WW/\$01D3000.FL1 -e && echo -e "\n\n\n EXTRACTING FL2 \n\n\n" && /usr/bin/python2.7 /usr/local/bin/uefi-firmware-parser ./app/G2ETB5WW/\$01D3000.FL2 -e --superbrute && echo -e "\n\n\n Testing me_cleaner against all extracted files.... output given only on successful files... \n\n\n" && find ./pfheader* ./capsule-Capsule/ ./volume-0/ -type f | while read filename; do python ~/me_cleaner/me_cleaner.py $filename &>/dev/null; if [ $? -eq 0 ]; then echo $filename; fi; done;

None of the commands above reports any matching me.bin probable file that me_cleaner would have been able to work on (first extracted by innoextract from exe, then FL1 and FL2 files being extracted per by uefi-firmware-parser).

Goal here would be to extract 8Mb SPI ROM from FL2, apply me_cleaner (python me_cleaner.py -S -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_down_dump.bin) to extract+neuter+deactivate ME and recreate expended IFD to be dropped in blobs/x230 to be usable by remote public CIs to produce reproducible builds from ROM upgrades downloaded directly from manufacturers, to mitigate legal issues and manual scripts like presently enforced from the blobs/boards directories.

Then the goal would be to extend to all boards requiring blobs.

@Mrchromebox? @osresearch? @theopolis? @merge @n4ru?

MrChromebox commented 4 years ago

I feel like this needlessly complicates the problem due to it's rigidity, fragility, and needless repetition of static tasks. If we wanted to have this as a manual script that feeds a blobs submodule, then I'd say fine, but do we really want to be downloading from Lenovo every simple time we build a Thinkpad board? And twice if building variants with and without a HOTP key? And how does this extend to other boards?

Purism originally did this with their coreboot build script, and ultimately there was lots of breakage due to sites outside of Purism't control going down, changing URLs, etc. Then we shifted to simply downloading/extracting from one of our own images. Currently I'm moving everything to a blobs submodule, because it's easier/cleaner for both us and our users.

tlaurion commented 4 years ago

@mrchromebox: the issue here is for x220 x230 t420 t430 and coming t530, all depending on Lenovo distribution licensing agreement to permit redistribution of ME blobs, which they haven't prohibited nor permitted.

Thinkpad-ec (interested in FL1 files, not FL2) project like many others, download iso (bootable dos) or img and patch ROM directly from constant Lenovo download URLs, which never changed.

Coreboot-blobs don't want to include new blobs without accompanying license from manufacturer. So from my understanding, Purism may be in a position of redistributing the blobs (in coreboot-blobs or elsewhere if they like), but there is no clear stance for Heads distributing Lenovo blobs and consequences without licensing are unknown. 1vyrain distributes 4mb modified rom files, working around the issue of redistributing the blobs (they don't need to) and not modifying ME/IFD (Can't be done without external first flash of modified IFD anyway).

I don't see any legal problem with this proposed solution (take original rom, extract 8Mb ROM, reconstruct IFD and neuter ME), while others are less clear, at least to me. @merge ?

For CIs, those binaries, like archive file or git archive downloaded by heads buildsystem, files would be downloaded once per initial build, then cached until module files point to newer versions, where newer archives and associated patches would then be applied and the build system would reuse cached files without redownloading them.

So only the first initial board build would download from Lenovo (newer ROM updates being linked to other locking problems requiring downgrading)

Of course if you have any information which would confirm the non-issue of redistributing ME neutered blobs and expended IFD descriptors directly under Heads, that would change the whole situation. Else it currently requires the blobs to be extracted from users laptop for other board configurations to apply the script, which leads to not being able to recreate reproducible board images with those blobs included.

If an external process permits to reproduce the same verification hash from an external source, we could also directly distribute the blobs IMHO, without having to have heads download from Lenovo. But the script provided in the blobs directory should permit the user to arrive exactly at the same hash measurements, without having to trust any third party other then the original manufacturer's released firmware.

@MrChromebox? @osresearch? @theopolis? @merge @n4ru? Any other idea?

tlaurion commented 4 years ago

Purism originally did this with their coreboot build script, and ultimately there was lots of breakage due to sites outside of Purism't control going down, changing URLs, etc.

@MrChromebox : Originally, Purism was downloading from mediafire/megadown for blobs that were extracted from flashed roms ( backuped, and then extracted); not original roms from original manufacturers which is a problem other boards legal issue/non-issue(still unclear...) are facing (maybe not Purism since manufacturer? IDK).

Then we shifted to simply downloading/extracting from one of our own images.

Exactly. That would be the equivalent of downloading original BIOS files from Lenovo/other Manufacturers (iso, exe), from original non-changing URLs, and applying recipes to be able to extract consistently blobs from those original images, but automatically and reproducibly for a specific original version (in PoC if working: 2.75 for X230) and applying maximal neutering+deactivation recipes from there to have maximal space usable in ROMs consistently.

Else like I said, we should have blobs distributed directly, with a script in the same directory (resolving license issue) permitting to arrive to the exact same result. That would be my way to address your concern of downloading from Lenovo (which is reliable as opposed to mediafire/megadown), if the tools to extract those blobs were working as intended.

Currently I'm moving everything to a blobs submodule, because it's easier/cleaner for both us and our users.

@MrChromebox : interested in that. Will you provide the blobs directly? What is your approach?

tlaurion commented 4 years ago

Trying the chipsec path:

wget https://download.lenovo.com/pccbbs/mobiles/g2uj31us.exe && innoextract ~/g2uj31us.exe && echo -e "\n\n EXTRACTING FL1\n\n\n" && echo -e "\n\n EXTRACTING FL1\n\n\n" && python chipsec_util.py -n uefi decode ./app/G2ETB5WW/\$01D3000.FL1 && echo -e "\n\n\n EXTRACTING FL2 \n\n\n" && python chipsec_util.py -n uefi decode ./app/G2ETB5WW/\$01D3000.FL2 && echo -e "\n\n\n Testing me_cleaner against all extracted files.... output given only on successful files... \n\n\n" && find ./app/G2ETB5WW/\$01D3000.FL*.dir/ -type f | while read filename; do python ~/me_cleaner/me_cleaner.py $filename &>/dev/null; if [ $? -eq 0 ]; then echo $filename; fi; done;

Same result: no match from me_cleaner.

n4ru commented 4 years ago

I’ll take a look at this. The offsets are identical generally across the entire generation, so once you find them for one chipset they’ll work to extract from the entire line.

I haven’t kept up entirely here, but I assume the end goal is to avoid legal issues by not directly hosting any proprietary code and instead downloading from Lenovo and extracting on the fly. I know the bios and ec can be extracted for certain, but am unsure about the ME and IFD. I’ll run some binwalks and toss the results into uefitool.

tlaurion commented 4 years ago

@n4ru: you got it right that is exactly the goal, having Lenovo images downloaded and 8mb firmware rom (including IFD, ME) extracted so that me_cleaner can be ran against it in the case of x230, and extending to other boards from there with fixed tools.

tlaurion commented 4 years ago

Functional test should be :

python me_cleaner.py -S -r -t -d -O unneeded_me.bin -D ifd_expended.bin -M me_shrinked.bin externally_extracted_rom.bin

Where 

ifd_expended.bin

and 

me_shrinked.bin

are the desired files to be included in coreboot config to build fully valid 12mb internally flashable ROM images, that can be cut in 4mb and 8mb flash images to be externally flashable too directly from CIs.

n4ru commented 4 years ago

Here you go.

ME Cleaner

Download and extract to the Drivers folder. In there, ME8_5M_Production.bin is the file. However, it is NOT a UEFI module, and I'm not sure if this is enough, as it is the binary itself, whereas normally UEFI modules have some "wrapping" when written to the BIOS chip itself.

Running it with -S gives the error

ME/TXE image detected -d, -D, -M, -S and -s require a full dump

Running unME11 on it gives - FPT.HeaderLength: Got [48], expected [32], but that's not surprising since it is ME 8.0 and not 11.

tlaurion commented 4 years ago

@n4ru :

ME/TXE image detected -d, -D, -M, -S and -s require a full dump

This is why finding a way to extract ME region from full ROM would be better, extracting minimal ME size and expended IFD in one shot, reproducibly.

tlaurion commented 4 years ago

@n4ru

Well, good news: wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O heads/blobs/x230/extracted_me.bin app/ME8_5M_Production.bin

ME/TXE image detected
Found FPT header at 0x10
Found 23 partition(s)
Found FTPR header: FTPR partition spans from 0x180000 to 0x24a000
ME/TXE firmware version 8.1.72.3002 (generation 2)
Public key match: Intel ME, firmware versions 7.x.x.x, 8.x.x.x
Reading partitions list...
 ???? (0x000003c0 - 0x000000400, 0x00000040 total bytes): removed
 FOVD (0x00000400 - 0x000001000, 0x00000c00 total bytes): removed
 MDES (0x00001000 - 0x000002000, 0x00001000 total bytes): removed
 FCRS (0x00002000 - 0x000003000, 0x00001000 total bytes): removed
 EFFS (0x00003000 - 0x0000df000, 0x000dc000 total bytes): removed
 BIAL (NVRAM partition, no data, 0x0000add0 total bytes): nothing to remove
 BIEL (NVRAM partition, no data, 0x00003000 total bytes): nothing to remove
 BIIS (NVRAM partition, no data, 0x00036000 total bytes): nothing to remove
 NVCL (NVRAM partition, no data, 0x00010511 total bytes): nothing to remove
 NVCM (NVRAM partition, no data, 0x0000493f total bytes): nothing to remove
 NVCP (NVRAM partition, no data, 0x0000a553 total bytes): nothing to remove
 NVJC (NVRAM partition, no data, 0x00004000 total bytes): nothing to remove
 NVKR (NVRAM partition, no data, 0x0001257d total bytes): nothing to remove
 NVOS (NVRAM partition, no data, 0x00034af7 total bytes): nothing to remove
 NVSH (NVRAM partition, no data, 0x00007609 total bytes): nothing to remove
 NVTD (NVRAM partition, no data, 0x00001eac total bytes): nothing to remove
 PLDM (NVRAM partition, no data, 0x0000a000 total bytes): nothing to remove
 GLUT (0x000df000 - 0x0000e3000, 0x00004000 total bytes): removed
 LOCL (0x000e3000 - 0x0000e7000, 0x00004000 total bytes): removed
 WCOD (0x000e7000 - 0x000140000, 0x00059000 total bytes): removed
 MDMV (0x00140000 - 0x000180000, 0x00040000 total bytes): removed
 FTPR (0x00180000 - 0x00024a000, 0x000ca000 total bytes): NOT removed
 NFTP (0x0024a000 - 0x0004a4000, 0x0025a000 total bytes): removed
Removing partition entries in FPT...
Removing EFFS presence flag...
Correcting checksum (0xed)...
Reading FTPR modules list...
 UPDATE           (LZMA   , 0x1cc508 - 0x1cc6c6       ): removed
 ROMP             (Huffman, fragmented data, ~2 KiB   ): NOT removed, essential
 BUP              (Huffman, fragmented data, ~56 KiB  ): NOT removed, essential
 KERNEL           (Huffman, fragmented data, ~135 KiB ): removed
 POLICY           (Huffman, fragmented data, ~91 KiB  ): removed
 HOSTCOMM         (LZMA   , 0x1cc6c6 - 0x1d343f       ): removed
 RSA              (LZMA   , 0x1d343f - 0x1d872a       ): removed
 CLS              (LZMA   , 0x1d872a - 0x1ddec0       ): removed
 TDT              (LZMA   , 0x1ddec0 - 0x1e45be       ): removed
 FTCS             (Huffman, fragmented data, ~18 KiB  ): removed
 ClsPriv          (LZMA   , 0x1e45be - 0x1e499f       ): removed
 SESSMGR          (LZMA   , 0x1e499f - 0x1f32cb       ): removed
Relocating FTPR from 0x180000 - 0x24a000 to 0xd00 - 0xcad00...
 Adjusting FPT entry...
 Adjusting LUT start offset...
 Adjusting Huffman start offset...
 Adjusting chunks offsets...
 Moving data...
The ME minimum size should be 98304 bytes (0x18000 bytes)
Truncating file at 0x18000...
Checking the FTPR RSA signature... VALID
Done! Good luck!

Good news:

user@heads-x230:~$ ls -al heads/blobs/x230/me.bin heads/blobs/x230/extracted_me.bin 
-rw-r--r-- 1 user user 98304 Mar 15 09:48 heads/blobs/x230/extracted_me.bin
-rw-rw-r-- 1 user user 98304 Feb 17 12:17 heads/blobs/x230/me.bin

Weird:

user@heads-x230:~$ sha256sum heads/blobs/x230/me.bin heads/blobs/x230/extracted_me.bin 
079b38c2ffaee7b0d170a10fafd802f25682bf8f0c10941f3744dfec57e6e0dc  heads/blobs/x230/me.bin
c140d04d792bed555e616065d48bdc327bb78f0213ccc54c0ae95f12b28896a4  heads/blobs/x230/extracted_me.bin

EDIT: not so weird since ME versions were not the same between sha256sums:

user@heads-x230:~/heads$ python ~/me_cleaner/me_cleaner.py -c blobs/x230/extracted_me.bin 
ME/TXE image detected
Found FPT header at 0x10
Found 1 partition(s)
Found FTPR header: FTPR partition spans from 0xd00 to 0xcad00
ME/TXE firmware version 8.1.72.3002 (generation 2)
Public key match: Intel ME, firmware versions 7.x.x.x, 8.x.x.x
Checking the FTPR RSA signature... VALID
user@heads-x230:~/heads$ python ~/me_cleaner/me_cleaner.py -c blobs/x230/me.bin 
ME/TXE image detected
Found FPT header at 0x10
Found 1 partition(s)
Found FTPR header: FTPR partition spans from 0xcc0 to 0xcacc0
ME/TXE firmware version 8.0.12.1498 (generation 2)
Public key match: Intel ME, firmware versions 7.x.x.x, 8.x.x.x
Checking the FTPR RSA signature... VALID

But MeAltDisable bit is not set...

-S, --soft-disable    in addition to the usual operations on the ME/TXE
                        firmware, set the MeAltDisable bit or the HAP bit to
                        ask Intel ME/TXE to disable itself after the hardware
                        initialization (requires a full dump)
  -s, --soft-disable-only
                        instead of the usual operations on the ME/TXE
                        firmware, just set the MeAltDisable bit or the HAP bit
                        to ask Intel ME/TXE to disable itself after the
                        hardware initialization (requires a full dump)
tlaurion commented 4 years ago

Here is what I propose. Please see README file. We could implement a module file that would extract me.bin automatically. No way to do the same for ifd.bin for the moment.

https://github.com/osresearch/heads/compare/master...tlaurion:x230_libremkey-repro-gitlab-circleci_blobs?expand=1

Resulting ROMs from artifact section here.

@osresearch @merge @kylerankin @MrChromebox @SebastianMcMillan Thoughts?

This ME blob can be used over: 

Helix (Type 3xxx)
T430, T430i, T430s, T430si, T431s
T530, T530i
W530
X1 Carbon (Type 34xx), X1 Helix (Type 3xxx), X1 Helix (Type 3xxx) 3G
X230, X230i, X230 Tablet, X230i Tablet, X230s
MrChromebox commented 4 years ago

@tlaurion so you're proposing to put the blobs in the main Heads repo, along with reproducibility instructions? Any reason to not put them in a submodule as was discussed previously?

tlaurion commented 4 years ago

@MrChromebox : ifd.bin still has to be extracted manually from an already flashed ROM image so solution is not complete as of now.

What I could do is have ifd.bin already there, and me.bin being extracted as a xx30-blobs submodule if specified in board config. That is the preferred way to go?

tlaurion commented 4 years ago

@MrChromebox : i'm confused. Thought you didn't want each build to download from Lenovo here