Consider including git for /boot and root fs integrity and changes reporting

[tlaurion]

Testing on top of qemu TCG with debian-12. Nonetheless to say, hashing on qemu TCG is turtle slow unfortunately but I do not see anything wrong.

[   70.903225] DEBUG: Root filesystem on /dev/vda3 lacks one of the configured directories: bin boot lib sbin usr
[   70.931377] TRACE: /bin/root-hashes-gui.sh(348): open_root_device
[   70.963936] TRACE: /bin/root-hashes-gui.sh(264): open_root_device_no_clean_up
[   71.001640] TRACE: /bin/root-hashes-gui.sh(232): open_block_device_layers
[   71.096543] TRACE: /bin/root-hashes-gui.sh(208): open_block_device_luks
[   81.971501] TRACE: /bin/root-hashes-gui.sh(232): open_block_device_layers
[   82.055642] TRACE: /bin/root-hashes-gui.sh(164): find_lvm_vg_name
[   82.279813] TRACE: /bin/root-hashes-gui.sh(187): open_block_device_lvm
[   82.476602] TRACE: /bin/root-hashes-gui.sh(232): open_block_device_layers
[   82.570460] TRACE: /bin/root-hashes-gui.sh(164): find_lvm_vg_name
[   82.751002] DEBUG: Did not detect an LVM2 PV: /dev/mapper/vgubuntu-root
[   82.822510] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[   82.924851] EXT4-fs (vda3): re-mounted. Opts: (null)
[ 1296.068011] EXT4-fs (vda3): re-mounted. Opts: (null)
[ 1296.152712] TRACE: Under /etc/functions:update_checksums

1996-82=1214 seconds = 20.2333 minutes Even if qemu TCG is not representative of real time performances, it shows computation weight of such process to hash root filesystem with default partitions, and shows a profound inneffectiveness, even if sound, process.

I keep having the same insight/intuition over and over about having a minimal git binary to not only detect and report on files changed, but also being able to report on what changed inside of those text files. It's more food for thought then anything else here.

A .git for /boot would be growing in time but nothing compared to root filesystem changes, and hashing its filesystem content would eventually become exponentially big to hash again from Heads, falling in the same pit. A hack around that could be to create a raw loop file for .git irectory, stored under /boot and that file only being hashed instead of .git directory content. That loop file, mounted and applied on top of tmpfs/ramfs at boot and replacing the current /boot / checks which I think would be faster.

Could even be a thin LVM at this point that would grow in time with usage, where git repo could also be wiped and recreated at moment of detach signing /boot content, with the sole purpose of actually tracking filesystem changes and report on them since they were detached signed. That might be more efficient then hashing whole filesystems. @JonathonHall-Purism good food for thought?

Originally posted by @tlaurion in https://github.com/linuxboot/heads/issues/1586#issuecomment-1921869907

[JonathonHall-Purism]

Yeah I thought something about this too since the question always goes:

• Heads: "something is modified"
• User: "what?"
• Heads: :shrug:

(I did not think of using git, that's a good thought.)

There's some value here but I also think the root question is really "Is that change intentional or tampering?" and it's going to be hard for many users to answer that question, even if they could actually see the changes. In some cases, and for technical Linux users, sure, sometimes it helps. But not everybody that needs Heads security has that knowledge, and often the changes are not easily reviewable (anything to do with binary files).

IMO, I think the real answer to "Is that change intentional or tampering?" is to eliminate the question - the OS could integrate with this system and sign its intentional /boot updates (which would require user's knowledge since they must provide the key). That's far from trivial and probably can't be required by Heads, but in the grand scheme of things I think it is the best path to providing tamper evident security to more users.

So this idea definitely has some value, it's just a question of how much and whether it is the best way to allocate time.

[tlaurion]

From really old experiences with bootstrapping minimal os (gentoo, never forget), we could leverage chroot and extend what you've done with rootfs checksumming to leverage distribution based tools and OS' packages signed databases.

Pseudocode but you'll get the gist.

Fedora

#!/bin/bash
# Mount the fs rootfs into a subdirectory of heads /
mount /dev/sdb1 /newroot

# Mount the necessary file systems and devices
mount -t proc proc /newroot/proc
mount -t sysfs sys /newroot/sys
mount -o bind /dev /newroot/dev
mount -o bind /run /newroot/run

# Change the root directory and run rpm -V
chroot /newroot /usr/bin/rpm -V -a -c -v
#above line to be tweaked to grep only checksum failing files, being config files or not

# Exit the chroot and unmount the file systems and devices
exit
umount /newroot/proc
umount /newroot/sys
umount /newroot/dev
umount /newroot/run

# Unmount and delete the fs rootfs
umount -l /newroot
rmdir /newroot

For debian, with needed additional debsums since not deployed by default

#!/bin/bash
# Mount the fs rootfs into a subdirectory of heads /
mount /dev/sdb1 /newroot

# Mount the necessary file systems and devices
mount -t proc proc /newroot/proc
mount -t sysfs sys /newroot/sys
mount -o bind /dev /newroot/dev
mount -o bind /run /newroot/run

# Change the root directory and run debsums -c
chroot /newroot /usr/bin/debsums -c
#above line to be tweaked, never played with debsums

# Exit the chroot and unmount the file systems and devices
exit
umount /newroot/proc
umount /newroot/sys
umount /newroot/dev
umount /newroot/run

# Unmount and delete the fs rootfs
umount -l /newroot
rmdir /newroot

@JonathonHall-Purism ? By mounting boot into rootfs we could verify most of the integrity checks provided by OS directly under Heads.

Limitations:

• initrd file is generated. No easy way to verify this. We could inspect policies and scripts that are used to generated the initrd.... But every OS varies here AFAIK which is why UKIs are desired and pushed forward
• grubenv is touched by OS. Not managed by package manager. No way to verify this. git would help advanced users evaluate changes.
• kernel, Xen, grub libraries/bin are versioned and verified. Wouldn't warn anymore, or, deeper check could be proposed. rpm qhery for file would point to package deploying the file, and rpm -V on that package would tell if content is unexpected.

When combined: os verification would skim changeset and verify integrity against OS signed databases (compromise there possible, we cannot tell). Then config files could be introspected by git. User would attest on config files changes and have to trust initrd files until unified kernels land which are supposed to pack and sign all of those including Linux command line options (I follow changes but not tightly : qubes os goes there as well as other OSes, we will need to tackle this at some point and fast when it happens, legacy boot is planned to be deprecated).

@JonathonHall-Purism unfortunately, unless Fedora silverblue or other immutable layers for OS, there is not much we can do here but inspect config files and trust the initramfs creation process. Thoughts?

[JonathonHall-Purism]

@tlaurion I think by chrooting into the OS you have a chicken-egg problem. You can't use the OS to decide if the OS is trustworthy.

I've pondered this a bit, and I still really think that working to eliminate this question in the first place is a better solution. While there could be some benefit to trying to add more information to the prompts in this state, I don't think it is a lot, and we also could easily overwhelm with information that still doesn't give a clear answer to the question of "is this tampering or intentional".

E.g. config file diffs (against either prior or packaged versions) do add something, but I think these most often occur as part of a larger set of updates, not a single change you can easily review. And most people are probably not familiar with the details of every config file, and there is still the problem that some influence initrds that then need to be checked, etc.

Signing the result of an update as part of the update would solve this much more neatly and thoroughly IMO. And we don't necessarily need support from distributions if we could provide hooks for their package managers, which is evidently possible for Debian, and probably others too. I opened #1615

[DemiMarie]

What about using dm-verity for /boot? /boot is small enough that one could simply have two different partitions and use dm-verity to authenticate it.

[tlaurion]

What about using dm-verity for /boot? /boot is small enough that one could simply have two different partitions and use dm-verity to authenticate it.

"small enough" is still 1gb, so x2 = 2gb and doesn't answer the question "are the changes legitimate"

[DemiMarie]

What about using dm-verity for /boot? /boot is small enough that one could simply have two different partitions and use dm-verity to authenticate it.

"small enough" is still 1gb, so x2 = 2gb and doesn't answer the question "are the changes legitimate"

My intention is that one does not modify /boot in-place at all, so there are never legitimate changes. Instead, one creates a new /boot and either signs the new root hash, writes it to a TPM NV index, or both. Heads would then pick the /boot to use based on either the NV index value or based on a generation number included in the dm-verity metadata.

This avoids a few problems with the current code:

1. It mounts a filesystem without any cryptographic protection of the filesystem’s integrity. Linux filesystem developers consider mounting an untrusted filesystem to be a bad idea, and don’t consider “malicious filesystem can cause code execution in the kernel” to be a security vulnerability. (For ext4 specifically, Chrome OS does consider this a security problem.)
2. There is no guarantee that modifications to /boot are atomic.

[tlaurion]

The root problem (what was changed since /boot was signed?) was raised again this morning under Heads matrix channel

Question:

Maybe this is more of a Qubes question but I just installed Qubes R4.2 on a machine, login first time via "boot options > ignore tampering" to set installation options like default qubes etc and afterwards turned off and turn on again to own the states. After re-ownership, I sign the contents of /boot and login to OS. I restart the machine without connecting to internet and verification of the boot content was successful. 2nd time, I turned on the machine and connected to internet, wait for 10 minutes and turned it off again. After this heads tells me that /boot/grub2/grubenv is changed. Any reason for such a change? I do not understand why grubenv didn't change just after the first restart and only changed after the second, when I connected to internet.

---

@JonathonHall-Purism : this wouldn't be answered by https://github.com/linuxboot/heads/issues/1615

@DemiMarie @marmarek:

It would be nice if you participated in those discussions.

They are aimed at making things better for the whole ecosystem which QubesOS is first candidate, but that still requires discussions prior of going into implementation. And we are not there yet because of lack of interaction. We have to agree on the source of the problem and pounder strategies prior of throwing ourselves into implementing something that might not fit the need. Differentiating need of strategies to fit the need is the important part here.

Please give input here. Heads is not qubesos, qubesos is not heads but they are used together. So let's start acting like it a bit more maybe?

I still think that #1599 and #1615 are complementary. Input needed please.

[JonathonHall-Purism]

:thinking: Does anybody know why Qubes changed grubenv on the second boot? It might be reproducible, maybe I can reinstall Qubes on a device and see if it happens.

I still think that #1599 and #1615 are complementary.

Ultimately I do agree with you. There's definitely some amount of value in being able to report precise differences in files when this happens.

I think that #1615 provides much more value - I think it could make Heads more usable for a wider audience, which is important to me. But #1599 definitely has some value too.

The question to me is - if we had shown the difference in grubenv here, would it have changed the user's experience in any way? Would they have known from the content of the file whether the change was OK or not? Some very technical users would, but many users would not. Maybe it would at least improve the help they got on the Matrix channel.

1599 won't help for OS updates, because they'll virtually always update kernels/initrds/other binary files. #1615 improves OS updates immensely by eliminating the question entirely.

So again I do think both have value. I think #1615 has much more value and could make Heads usable to a wider audience, but #1599 still offers some benefit.

[JonathonHall-Purism]

:thinking: Thinking about implementation here (since it's the title of the issue :wink:) I think that using git for this would be overkill. I think we would be better off just tar'ing up all of /boot when it's being signed and including that in the signatures. That way it's easy to validate that it's signed before attempting to use it.

[tlaurion]

:thinking: Thinking about implementation here (since it's the title of the issue :wink:) I think that using git for this would be overkill. I think we would be better off just tar'ing up all of /boot when it's being signed and including that in the signatures. That way it's easy to validate that it's signed before attempting to use it.

@JonathonHall-Purism that wouldn't answer the "what changed" doesn't it?

Alternative would be to keep tarball, I agree, where we have no control here on what the end user puts under /boot, where a git init there, then producing digest including .git and signing it would resolve the "what changed" in introspective goal. Then wrapping on git being clean/unclean and then giving possibility of wrapping on git fiff output would resolve it all without reinventing the wheel.

Before, some wrapping was done against diff output (in luks header) but that was too cryptic and would require to keep tarball, decompress, then call and wrap on diff.

Git is actually the only effective thing I have in mind, which is why title of this issue is implementation goal challenge, and configuring it for minimal usage for compilation would be expected to not be too big here.

Am I missing something?

[JonathonHall-Purism]

@JonathonHall-Purism that wouldn't answer the "what changed" doesn't it?

Yes it would. You'd untar the signed copy and diff it against whatever is currently on /boot. You'd know the tarball is safe to use because it's still signed.

Before, some wrapping was done against diff output (in luks header) but that was too cryptic and would require to keep tarball, decompress, then call and wrap on diff.

If doing a recursive diff ourselves is impractical, we could in principle use git just for its diffing abilities (only store a tarball, but then unpack it into a clean git repo, commit, copy the current /boot, git diff, etc.) But I think we could get what git does just by invoking diff the right way.

Am I missing something?

I think leaving a git repo on /boot leaves open too many vectors for somebody tampering /boot to gain code execution in Heads via git. A signed tarball is really easy to validate, a git repo isn't, and I don't see any benefit to storing a git repo. Signing the commits is not enough, git will still read plenty of repo metadata/config/etc. even if the commits aren't signed, and those are potentially malicious in this scenario.