Open tlaurion opened 1 year ago
hi @tlaurion
I would advise into asking specific questions in the issue I created as raw notes for integrity validation, which hopefully will result into an additional wiki page to answer all those questions.
okay, since we want to create additional wiki page, then i will make conclusion here, from https://github.com/osresearch/heads-wiki/issues/109 https://github.com/osresearch/heads-wiki/issues/110 https://github.com/osresearch/heads-wiki/issues/111 https://github.com/osresearch/heads-wiki/issues/112 https://github.com/osresearch/heads-wiki/issues/113 https://github.com/osresearch/heads-wiki/issues/114
All commits are signed, before being pushed into repository,
then being reviewed, before merging, & tested by regression testing,
This is how to distrust the infrastructures & ensure integrity,
across network, repository, Circle CI, local compile machine, & flashing tool.
2 ways, it verifies authenticity & integrity, prior compiling:
If Heads' code base, related to specific commit, is modified, prior compiling, either Circle CI compiling, or Local compiling, then it will produce ROM, & "-dirty" will be appended to the ROM name, to notify user that the ROM build is not clean, otherwise, if ROM name contains no "-dirty", then we can assume it is clean.
Commit ID will be appended to the ROM name, as an information, that the ROM was built, by using what commit.
hashes,txt containing hashes of built files & ROM, will be generated after built, & can be used to verify integrity, for 3 conditions:
hashes.txt generated, after built, will only contain hashes for built files & ROM, it will not contain any hashes for any files before built.
Currently, Heads has reproducibility issues, which mean, using different machine, to build / compile Heads, i.e. using Circle CI or Debian laptop or docker, will produce ROM with different hash / checksum.
hashes.txt will show individual hashes, of everything packed in the firmware, and also the firmware image itself. If final hash of final ROM image is same, it means all parts inside are good.
Compressed under initrd.cpio.xz:
hashes.txt under ./blobs/ section, are different from hashes.txt, generated during compiling:
Blobs (ME, GBE, IFD) will have different hashes, they are unique for each laptop:
By using blobs (ME, GBE, IFD) from Heads repository, then we will have same blobs & its hashes, & also same MAC address, as other Heads' user. But using your own blobs, will have unique MAC address & motherboard info, etc.
Flashed Heads ROM, will contain MRC cache, GPG public key ring, & config.user override. Therefore, while backup-ed, will have different hash checksum, so need to extract / unpack 1st, then validate extracted component with hashes.txt, of same commit.
Re-flashing / Updating Heads externally, provide more integrity, than internally re-flashing / updating, but for most cases, is not necessary. Internal flashing rely on internal flashrom tool, in case of distrusting the internal flashrom tool, then can do external re-flashing. Re-flashing externally will invalidate measurements, since the ROM won't contain the same GPG key ring. The risk of flashrom being compromised is low, but possible.
Internal flashing, rely on internal flashrom tool. sha256sum -c is a subprogram of busybox, to check digest integrity.
After re-flashing, check integrity internally, by using sha256sum of internal busybox, and take backup internally / externally, & unpack / extract for external re-check.
From top post
After our binwalk --extract command:\n\nuser@heads-backup-extraction:/tmp/inspection/payload_content$ cd _payload.extracted/\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls\n2C9B80.xz 2E531F 2E531F.xz 32D5 32D5.xz\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ wget -O extract-vmlinux https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ chmod +x ./extract-vmlinux \nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ls -al\ntotal 47316\ndrwxr-xr-x 2 user user 4096 Nov 25 12:34 .\ndrwxr-xr-x 3 user user 4096 Nov 25 12:22 ..\n-rw-r--r-- 1 user user 4405151 Nov 25 12:22 2C9B80.xz\n-rw-r--r-- 1 user user 12882944 Nov 25 12:22 2E531F\n-rw-r--r-- 1 user user 4292608 Nov 25 12:22 2E531F.xz\n-rw-r--r-- 1 user user 19528104 Nov 25 12:22 32D5\n-rw-r--r-- 1 user user 7315530 Nov 25 12:22 32D5.xz\n-rwxr-xr-x 1 user user 1695 Nov 25 12:34 extract-vmlinux\n\nuser@heads-backup-extraction:/tmp/inspection/payload_content/_payload.extracted$ ./extract-vmlinux 32D5 > vmlinux_local\n
This is coming up again and again and again: how to (reasonably) trust USB Security Dongles's flashing green light (reverse HOTP, Heads compatible dongle) / TOTP matching scanned Qr code at TOTP sealing / that TPM Disk Unlock Key passphrase isn't typed/captured in/from compromised environment
Note that here, cbmem -L
under recovery shell will output coreboot measurements (event log) for each measured boot part which encompasses bootblock up to payload.
The technical coreboot doc is here https://doc.coreboot.org/security/vboot/measured_boot.html
Coreboot creates an event log that can be audited through cbmem log and specifically through cbmem -L
. But that is produced/reported by the same authentic/potentially compromised stages which could be crafted to report fake measurements (theoretically, not practically proven as of now, but still...)
And again, payload being measured and sealed, by Heads, into TOTP/HOTP secret, takes care of the "meta measures" sealed from PCR 0-4 into a secret (TOTP) that can be unlocked only if the same sealed/unsealed PCR measurements are as intended (matching event log). But how to trust that the coreboot phases are not lying on recalculated reported values? We cannot reasonably trust bootblock measurement from hypothetical possible attacks.
Then Heads extends those even log measurements with runtime information, including what is seen here as "user stuff" early at boot : key material, config user etc, where going to recovery shell also modifying tpm PCRs and preventing unsealing of secrets.
Same for TPM DUK, which add up to what is measured and sealed. Including loaded on demand kernel modules and LUKS header. All documented at https://osresearch.net/Keys/#tpm-pcrs
To see those in action, one would have to activate debug/tracing in recent ROM images from config settings menu and save in rom, reseal, signing and follow onscreen instructions to reseal TPM DUK if that feature is activated when setting a default boot configuration.
But instructions on top post permits to validate things only manually. Could we automate another layer of verification that would require other layers to be compromised as well for an attack to succeed? Yes, we could.
Heads could add an additional check. bootblock could be extracted (cbfs tool from flashtool) and hashed in ram(sha256sum from busybox) and be part of detached signatures (gpg tool from gnupg) under /boot as additional crosscheck, since the paranoia going on, untrusting bootblock not faking measurements for itself and next stages is still present... And bootblock being our root of trust in lack of an hardware root of trust.
Alternative
So we could extract bootblock (cbfs being compiled from modules/flashtools not coreboot) and hash it in memory (modules/busybox sha256sum) and have those hashes as part of the detached signed digest signed and verified by gpg (modules/gnupg2).
We would add an additional delay to verify bootblock prior of trusting Heads and coreboot measured boot alone.
Recap. Measured boot from coreboot starts by bootblock enabling tpm. And then having bootblock measuring itself and reporting its own measurement to extend TPM PCR 2. And then bootblock measures next stage, extends PCR 2 again, and so on with other stages up to payload(heads kernel+initrd) . So compromising bootblock would definitely changes its checksum. And not trusting bootblock measurement through coreboot measured boot alone could be an improvement, considering that not only coreboot would need to be compromised but also Heads.
Going back in previous notes of this thread, not only cbfs would need to be tampered with in cbfs, but payload as well, to compromise also busybox, flashtools and gnupg2 modules which hashes would change (output at build time not yet verified on boot path but could also be done!)
Thoughts?
Should be considered under #62
What exactly is the threat model here to defend against? Somebody modifying bootblock to lie about its measurement (and also measurements of next stage)? Modifying how? From recovery shell, flashrom from within the OS, external programmer?
Saving bootblock separately, to be dumped into /boot and signed makes such attack only slightly more complicated - instead of just saving "proper" hash, attacker would need to keep also the original bootblock, or at least parts that were changed. But since the attacker would modify coreboot payload anyway (after all, that's where DUK could be captured, not in the bootblock itself), faking also bootblock verification there would be quite simple (for example, modify cbfs to dump saved copy instead of the actual bootblock). Without hardware root of trust, it's hard to prevent such attack completely. One could detect it with external programmer by dumping flash content and comparing, but it isn't practical for everyday use.
One thing that could raise the bar for such attack is to write-protect bootblock in SPI flash. If done properly, it would force the attacker to use external programmer, which takes significantly more time (depending on hardware, sometimes just few screws, sometimes disassembly of most of the laptop). Case opened can be detected, for example with glitter on screws. Then updates would be more complicated, but that could be solved with either/and:
The last one is IMO desirable anyway, but it's a lot of work too. And there still needs to be some other update method, for example when switching between different build origins (signed with different keys).
(*) I write above "grant write access", but depending on technical details it could also mean "not protecting the area" under some conditions (as in: may require reboot into special mode, instead of reversing protection).
The technical coreboot doc is here https://doc.coreboot.org/security/vboot/measured_boot.html
This still could use some rework, I didn't do a good job reviewing it earlier. For example:
PCR-2 - Hash of Root of Trust for Measurement which includes all stages, data and blobs.
There is always only one entity that is the Root of Trust (although there may be more than one root/chain). It is implicitly (usually, unfortunately) trusted. It can be in hardware (Intel Boot Guard, AMD Hardware Validated Boot or whatever they're called today) or in software (initial part of firmware, e.g. coreboot bootblock). It is generally believed that hardware root of trust is safer, however in practice "hardware" RTM is usually just a software running on another chip, hopefully protected at least by RTV. "All stages, data and blobs" cannot be all part of root, otherwise you wouldn't have to measure anything, since by definition you trust the root.
theoretically, not practically proven as of now, but still...
Don't tempt me :slightly_smiling_face:
And again, payload being measured and sealed, by Heads, into TOTP/HOTP secret, takes care of the "meta measures" sealed from PCR 0-4 into a secret (TOTP) that can be unlocked only if the same sealed/unsealed PCR measurements are as intended (matching event log).
This seems a bit backwards. Event log is only informational, it can't be trusted unless PCR values are correct (assuming that the chain of trust wasn't broken at any point since the respective root was obtained). One improvement I can think of is to somehow convey the expected PCR values (through a different channel than the binary) and compare them with actual values. They are long, checking them manually would be prone to errors, but it is still hard to get similar, but not identical hash so maybe this would work in practice. Perhaps a mobile validator app could scan the QR code and compare that to make it easier for users, but I don't know if all important PCR values would fit on it, even in binary form.
But how to trust that the coreboot phases are not lying on recalculated reported values? We cannot reasonably trust bootblock measurement from hypothetical possible attacks.
This is why you have to implicitly trust the root. There are ways to make it more trustworthy, e.g. by physically making bootblock read-only with WP pin or by using hardware RoT. In any case, you still trust that those mechanisms actually work as advertised, that they can resist attacks, that the underlying math isn't broken. If the root is broken, whole chain is. Adding more links to that chain won't improve its security, it can only make it worse.
Heads could add an additional check.
If Heads is started by compromised earlier stages, how can you trust it to be the proper Heads, without this check tampered with?
@marmarek @krystian-hebel : Decided to go with WiP/Poc code to show what is attempted to be done here at https://github.com/linuxboot/heads/pull/1568#issue-2061148875
@marmarek @krystian-hebel : PoC and output, leading base to measured boot introspection and paving the way to forward sealing at https://github.com/linuxboot/heads/pull/1568#issuecomment-1874938170 and next comment
Flashkeeper project accepted by NLnet : https://nlnet.nl/project/Flashkeeper/
Finishing MoU, will keep you all posted with details, and will present at QuebesOS mini-summit 2024.
This is raw notes. This will get edited multiple times prior of having a base to create a wiki page.
The quick way, no-brainer, is to reflash the same downloaded/compiled firmware and keeping settings, as already documented at: https://osresearch.net/Updating#reflashing-the-same-firmware
Unless flashrom itself was compromised (meaning that the payload is tampered), reflashing the same firmware image using the menu options to flash new firmware while keeping settings will reflash IFD+ME+GBE+coreboot+payload+current CBFS files. Since coreboot measures its bootblock + ramstage+romstage+payload, and then Heads measures added CBFS content, TOTP/HOTP measurements should be the same, and TOTP/HOTP unsealing of secrets should match.
But that doesn't answer the introspection need of some users to actually validate that the firmware components are actually in the expected state. And those notes, even better, if translated into code, should answer those needs.
Taking an internal backup of the firmware to be inspected externally
From Heads Recovery console, one can easily obtain actual BOARD_NAME and FW_VER that includes the bits we want to differenciate properly the actual board and flashed Heads commit ID in case we want to go back.
From Heads recovery shell, with a USB thumb drive ready to receive backup, we mount usb drive in read+write and we take a backup with proper naming scheme:
Extracting content from that firmware image for validation
Unfortunately, coreboot-utils are precompiled for a limited number of distributions today. I'll take for granted here that debian12 is installed under QubesOS (see unman's repo https://qubes.3isec.org/Templates_4.1/) Then a quick
sudo apt install coreboot-utils binwalk
will make the tools available with/without sudo.Each Heads board's build comes with a hashes.txt file. It is either dowloadable from CircleCI artifacts, or can be seen, and copy pasted from a build's output given on screen. Following download instructions will lead you to the hashes step of any CircleCI build: https://osresearch.net/Downloading
Now. After having mounted our USB thumb drive under a debian-12 based qube with tools installed, from command line, having passed the USB thumb drive to our newly booted qube:
And then we work on our copied to memory image.
We can extract partitions not managed from coreboot actual versions measurements:
Here one could compare ME against what is downloaded and extracted from Heads scripts, GBE and IFD configs against what is stored under Heads tree.
But most questions are related to the firmware integrity itself, Heads itself:
Here again, a lot of stuff there.
Get build hashes for current rom
x230-hotp-maximized-Heads-v0.2.0-1296-g139ecb8_backup.rom is commit 139ecb8 We go over https://github.com/osresearch/heads/commit/139ecb8 We click grewn mark, follow the rabbit to the build for x230-hot-maximized at https://app.circleci.com/jobs/github/osresearch/heads/5448, click the "Output hashes" if this is an old build without artifacts, and copy the output of the hashes there to a local hashes.txt file we make sure to have available for our situation. We can directly save the output from CircleCI's Download Icon, which in our case leads to https://circleci.com/api/v1.1/project/github/osresearch/heads/5448/output/104/0?file=true&allocation-id=63751e76e6fb893f12bf3473-0-build%2F97AF006
From now on I consider we have that file saved into hashes.txt to be used locally.
Extract content from coreboot's payload
We now have coreboot's payload.
We extract the content with binwalk
So we have a matching initrd.cpio.xz into 2E531F.xz We could dig more into that, and will (after all, everything hashes.txt is detailing why we have a match for the payload).
hashes.txt can be used under recovery shell to verify hashes of the initrd binaries
The paths undr hashes.txt are relative. If we have hashes.txt under USB thumb drive, from Heads recovery shell: cd / mount-usb sha256sum -c /media/hashes.txt
Will give OK reports for all files that were found, including libraries and binaries and scripts user by heads, as seen under
Check bzImage
Unfortunately, this requires CircleCI bzImage since mathing can only be made against decompressed binary.
After our binwalk --extract command:
Download bzImage from CircleCI artifact