Initial integration of authenticity (fs-verity) capability into the system (disabled atm)

[rborn-tx]

This initial integration covers:

• Enabling commit signing in ostree;
• Generating the keys for signing as part of the build process;
• Adding the composefs image digest to the commit metadata and signing it;
• Saving the verification (public) key into the ramdisk;
• Ensuring fsverity is enabled on the repository upon boot;
• Asking ostree to check the deployment about to be booted with the key.

Please refer to the commit messages for further details.

Note: The commits are intentionally referring to an internal ticket (TOR-3379) because I didn't want GitHub to close #46 yet (until we actually finish the integration, which requires the kernel patches for fsverity).

[rborn-tx]

@MingliuYan I've added you as reviewer particularly because the work here has a lot of overlap with https://github.com/torizon/meta-toradex-torizon/pull/45. So I thought it'd be important for you to know about it besides having your review. Please take a look at the commit message titled "initramfs-framework: automatically enable fsverity on first boot" where I explained what I believe to be the advantages/disadvantages of the approach being followed with respect to fsverity.

[rborn-tx]

@jsrc27 fyi, in the latest commits I implemented a progress bar showing on the console the progress of the fsverity enabling process. This was done to (partially) fulfill the requests I got from internal feedback at Toradex.

[jsrc27]

fyi, in the latest commits I implemented a progress bar showing on the console the progress of the fsverity enabling process. This was done to (partially) fulfill the requests I got from internal feedback at Toradex.

Will this be the only addition from the feedback for now? Or are you planning to add more to this?

Also adding @EdTheBearded to the reviewers so he can take a look at this if he wants.

[EdTheBearded]

Everything seems to be ok, the only thing we might want to clear is this boot time increase. As I understood, this is just on first boot right? Subsequent boots have no increase, or at least not an apparent one?

I really don't see that as a big issue, since it is the first boot only. But a PM opinion on this is valid, and maybe a note on the documentation as well.

[rborn-tx]

@EdTheBearded

Everything seems to be ok, the only thing we might want to clear is this boot time increase. As I understood, this is just on first boot right? Subsequent boots have no increase, or at least not an apparent one?

Yes, it's just the first boot after installing the OS.

... But a PM opinion on this is valid, and maybe a note on the documentation as well.

I've already asked for feedback internally at Toradex and got plenty of it actually. In fact, the progress bar was added as a result of that. An improvement that I proposed was adding feedback via plymouth as well, but that was left for the future. The overall opinion was that the time increase is not a problem as long as we document it well and provide user feedback, so I think we're all aligned.

[ldts]

@rborn-tx so enabling fs-verity during initramfs means that the fs-verity digests wont be taken from the signed composefs image during mount (which I suppose does not open to an attack vector?)

[rborn-tx]

@ldts

@rborn-tx so enabling fs-verity during initramfs means that the fs-verity digests wont be taken from the signed composefs image during mount (which I suppose does not open to an attack vector?)

Enabling fs-verity on a file means computing its Merkle tree (a tree of digests/hashes) and storing it somewhere in the filesystem. When we do this operation in the ramdisk we don't change the existing composefs image (part of a ostree deployment) at all; the existing composefs image will contain the "expected" digests of all files and when a file is accessed (via the mounted composefs image) its "measured" digest will be checked against the "expected" one in the composefs image. So, everything should be protected as long as the composefs image itself is protected against modifications.

An important follow up question you might ask is: "How is the composefs image protected?". The answer is: the image is also protected by fs-verity but the expected digest of that file is stored as metadata in a signed ostree commit. Note that the expected digest is determined at build time on a developer machine where the commit is also signed.

It's interesting to point out that the code in the ramdisk is also enabling fs-verity on the composefs image, but this is not a security issue because the "measured" digest of this file will be checked by ostree against the "expected" digest stored in a signed ostree commit (if I got it correctly from the ostree code, this check is done when mounting the composefs image).

[rborn-tx]

@jsrc27 @EdTheBearded Is there anything missing before we can merge this PR?

[ldts]

Enabling fs-verity on a file means computing its Merkle tree (a tree of digests/hashes) and storing it somewhere in the filesystem. When we do this operation in the ramdisk we don't change the existing composefs image (part of a ostree deployment) at all; the existing composefs image will contain the "expected" digests of all files and when a file is accessed (via the mounted composefs image) its "measured" digest will be checked against the "expected" one in the composefs image. So, everything should be protected as long as the composefs image itself is protected against modifications.

This is the bit I find confusing - since fsverity is not available during build (otherwise we would not be enabling it in initramfs) how are the expected measured digests ever reaching the composefs image?

The way I understood the kernel code, using the ramdisk to enable fs-verity on every file (along with the overlay 'metacopy' attribute) causes the measured digest at enable time to be copied to somewhere (not sure where, I guess ext4). Later on, on the first read, the measured digest gests compared against the overlay metacopy.

https://docs.kernel.org/filesystems/overlayfs.html#metadata-only-copy-up https://elixir.bootlin.com/linux/latest/source/fs/overlayfs/util.c#L1428

root@intel-corei7-64:/var/rootdirs/home/fio# findmnt /
TARGET SOURCE  FSTYPE  OPTIONS
/      overlay overlay ro,relatime,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on,verity=require

[rborn-tx]

@ldts

This is the bit I find confusing - since fsverity is not available during build (otherwise we would not be enabling it in initramfs) how are the expected measured digests ever reaching the composefs image?

The expected digests are determined by libcomposefs and they don't require fs-verity support by the kernel (on the build machine); that library has code to determine the digests and they match what the kernel would compute.

In our case the OS installer image is basically a tarball containing the rootfs which is unpacked by our installer on each device at installation time; in this process any fs-verity information is lost. That's why we re-enable fs-verity on each file; it's not because fs-verity wasn't enabled at build time.

In the end keeping fs-verity info is not strictly needed as long as you can reproduce it and that's what we are doing. By the way, rebuilding that info is a use case foreseen by the ostree guys which motivated the addition of the ostree admin post-copy command:

https://github.com/ostreedev/ostree/pull/3094

The way I understood the kernel code, using the ramdisk to enable fs-verity on every file (along with the overlay 'metacopy' attribute) causes the measured digest at enable time to be copied to somewhere (not sure where, I guess ext4). Later on, on the first read, the measured digest gests compared against the overlay metacopy.

Keep in mind that for composefs to work with fs-verity there were changes made on the kernel side that were implemented by the ostree/composefs guys; I haven't looked at them yet to understand what exactly they do but from the various comments on the ostree Git repo I understand they involve integration between erofs, overlayfs and fs-verity. A kernel without those integrations already allows composefs to work but without the fs-verity part.

[ldts]

The expected digests are determined by libcomposefs and they don't require fs-verity support by the kernel (on the build machine); that library has code to determine the digests and they match what the kernel would compute.

ahh yes you are right, I remember that code now. yes, so it all makes sense. perfect. thanks!