Add [dm|fs] verity support to puzzlefs

[ariel-miculas]

We need some research into https://github.com/containers/composefs first and then reason how to go about this.

[ariel-miculas]

From: https://lwn.net/Articles/917416/

As I said above, I don't think fs-verity actually works if underlayfs doesn't support fs-verity, which also includes one of composefs example -- FUSE.

From: https://www.spinics.net/lists/linux-fscrypt/msg06900.html

Although FUSE lacks the support of "unrestricted" ioctl, which makes it impossible for the filesystem to receive the fs-verity ioctls. Same to statx. I think that's where we'd need a change in FUSE protocol.

From https://github.com/containers/composefs:

Composefs also supports fs-verity validation of the content files. When using this, the digest of the content files is stored in the image, and composefs will validate that the content file it uses has a matching enabled fs-verity digest. This means that the backing content cannot be changed in any way (by mistake or by malice) without this being detected when the file is used. You can also use fs-verity on the image file itself, and pass the expected fs-verity digest as a mount option, which composefs will validate. In this case we have full trust of both data and metadata of the mounted file. This solves a weakness that fs-verity has when used on on its own, in that it can only verify file data, not metadata.

[ariel-miculas]

Background

From https://www.kernel.org/doc/html/next/filesystems/fuse.html

fuseblk
The filesystem is block device based. The first argument of the mount system call is interpreted as the name of the device.

From https://manpages.ubuntu.com/manpages/xenial/man8/mount.fuse.8.html

blkdev Mount a filesystem backed by a block device.  This  is  a  privileged  option.  The
              device must be specified with the fsname=NAME option.

From https://unix.stackexchange.com/a/471384

FUSE: From the kernel's point of view, this is backed by a userspace program. This program may in turn use a block device (fuseblk), NTFS is implemented with fuseblk. It may also use the network or anything else to present a file system.

Setup

$ dd if=/dev/zero of=backing-fs bs=4K count=4000
$ sudo losetup --find --show backing-fs
/dev/loop1
$ sudo mkfs -t ext4 -F -b4096 /dev/loop1

puzzlefs mount (see notes):

sudo target/debug/puzzlefs mount -f -o blkdev,fsname=/dev/loop1,allow_other /tmp/oci-simple first_try /tmp/puzzle

Conclusions

Using fuseblk requires:

• privileged operation
• an existing block device

So it doesn't seem we could use dm-verity with puzzlefs for now.

Other options: buse? (it's experimental and not production-ready) ublk? (new user-space block driver based on io_uring)

Notes

Patch to fuser

fuser running as root will bypass the fusermount call, this patch will prevent it from doing it

diff --git a/src/mnt/fuse_pure.rs b/src/mnt/fuse_pure.rs
index ea366df..31037c6 100644
--- a/src/mnt/fuse_pure.rs
+++ b/src/mnt/fuse_pure.rs
@@ -84,13 +84,13 @@ fn fuse_mount_pure(
         return fuse_mount_fusermount(mountpoint, options);
     }

-    let res = fuse_mount_sys(mountpoint, options)?;
-    if let Some(file) = res {
-        Ok((file, None))
-    } else {
+    // let res = fuse_mount_sys(mountpoint, options)?;
+    // if let Some(file) = res {
+    //     Ok((file, None))
+    // } else {
         // Retry
         fuse_mount_fusermount(mountpoint, options)
-    }
+    // }
 }

 fn fuse_unmount_pure(mountpoint: &CStr) {

dmverity on block devices:

#!/bin/bash
set -e
set -x

backing_file=fake-fs
devname=ariel-device

dd if=/dev/zero of="$backing_file" bs=4K count=4000
loopback_dev=$(sudo losetup --find --show "$backing_file")
sudo mkfs -t ext4 -F -b4096 "$loopback_dev"
sudo mount "$loopback_dev" /mnt
echo 'ana are mere' | sudo tee -a /mnt/foo
echo 'dar nu are pere' | sudo tee -a /mnt/bar
sudo umount /mnt
root_hash=$(sudo veritysetup format "$loopback_dev" mydev.hash | grep "Root hash:" | awk '{print $NF}')
sudo veritysetup open "$loopback_dev" "$devname" mydev.hash "$root_hash"
sudo mount /dev/mapper/"$devname" /mnt

[ariel-miculas]

Proposal for fs-verity:

1. puzzlefs build generates an array of [blob1-shasum, fs-verity-root-hash] and appends it to the puzzlefs image manifest (the file which index.json points to and which has media_type = application/vnd.puzzlefs.image.rootfs.v1).
2. puzzlefs enable-verity will be a new subcommand which compares the measured fs-verity root hash of the image manifest with the one passed on the command line, enables fs-verity on the image manifest, reads the array of [blob1-shasum, fs-verity-root-hash] and enables fs-verity for each file in the array.
3. puzzlefs mount will take another argument which will dictate how to use fs-verity (enabled or disabled); if it's enabled, then it will also take an additional fs-verity-root-hash argument which contains the root hash of the puzzlefs image manifest, issue an ioctl requesting the fs-verity root hash of the image manifest, compare the two and exit if there's a mismatch; puzzlefs mount will also issue an ioctl requesting the fs-verity root hash of each blob that it opens and it will compare it to the fs-verity-root-hash stored in the image manifest; if it doesn't match, then it will return an error (TBD: if we want to exit the process instead or make this behavior configurable).

[ariel-miculas]

fsverity configuration

Check fsverity feature in the kernel:

$ rg -z CONFIG_FS_VERITY /proc/config.gz
9987:CONFIG_FS_VERITY=y
9988:# CONFIG_FS_VERITY_DEBUG is not set
9989:CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y

Enable fs-verity on ext4 filesystem:

ariel@selected-chipmunk:~$ mount | grep ext4
/dev/sda1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
ariel@selected-chipmunk:~$ sudo tune2fs -O verity /dev/sda1
tune2fs 1.46.5 (30-Dec-2021)
ariel@selected-chipmunk:~$ sudo tune2fs -l /dev/sda1 | grep -i verity
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum verity

Enable fsverity on a file:

echo lorem ipsum > immutable
# get the hash without enabling fsverity
ariel@selected-chipmunk:~$ fsverity digest immutable
sha256:fda0e2d973b5c81e4a8ccc43a0bb13cbc9b5da8d83c4ff4416f6860ea20df0d6 immutable
# enable fsverity
ariel@selected-chipmunk:~$ fsverity enable immutable
# get the root hash
ariel@selected-chipmunk:~$ fsverity measure immutable
sha256:fda0e2d973b5c81e4a8ccc43a0bb13cbc9b5da8d83c4ff4416f6860ea20df0d6 immutable

Test fsverity:

ariel@selected-chipmunk:~$ echo dolor sit amet >> immutable
-bash: immutable: Operation not permitted

Overwrite the file by modifying the blocks directly:

ariel@selected-chipmunk:~$ sudo debugfs -R "stat /home/ariel/immutable" /dev/sda1
Inode: 260139   Type: regular    Mode:  0664   Flags: 0x180000
Generation: 2401168620    Version: 0x00000000:00000001
User:  1001   Group:  1001   Project:     0   Size: 12
File ACL: 0
Links: 1   Blockcount: 16
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x63ff2716:dbec86d8 -- Wed Mar  1 10:21:10 2023
 atime: 0x63ff26f0:10cff610 -- Wed Mar  1 10:20:32 2023
 mtime: 0x63ff2716:dbec86d8 -- Wed Mar  1 10:21:10 2023
crtime: 0x63ff26d6:e1b44e64 -- Wed Mar  1 10:20:06 2023
Size of extra inode fields: 32
Inode checksum: 0x2d3e414c
EXTENTS:
(0):394785, (16):25551

Get block size:

ariel@selected-chipmunk:~$ sudo dumpe2fs -h /dev/sda1 | grep "Block size"
dumpe2fs 1.46.5 (30-Dec-2021)
Block size:               4096

Check that we can read the contents of the file by reading directly from the disk:

ariel@selected-chipmunk:~$ sudo dd if=/dev/sda1 of=success.txt bs=1 count=12 skip=$((4096*394785))
10+0 records in
10+0 records out
10 bytes copied, 0.000388477 s, 25.7 kB/s
ariel@selected-chipmunk:~$ cat success.txt
lorem ipsum
ariel@selected-chipmunk:~$

Now modify the file by writing directly to the disk:

ariel@selected-chipmunk:~$ sudo dd of=/dev/sda1 if=/dev/zero bs=1 count=1 seek=$((4096*394785))
1+0 records in
1+0 records out
1 byte copied, 0.00350661 s, 0.3 kB/s

And check that the first byte is 0:

ariel@selected-chipmunk:~$ sudo dd if=/dev/sda1 of=success.txt bs=1 count=12 skip=$((4096*394785))
12+0 records in
12+0 records out
12 bytes copied, 0.000639262 s, 18.8 kB/s
ariel@selected-chipmunk:~$ hexdump -C success.txt
00000000  00 6f 72 65 6d 20 69 70  73 75 6d 0a              |.orem ipsum.|
0000000c

Now check that we can't open immutable:

ariel@selected-chipmunk:~$ cat immutable
lorem ipsum

Huh? Oh, we need to clear the page cache:

sync; echo 1 | sudo tee /proc/sys/vm/drop_caches

Now we shouldn't be able to open the file:

ariel@selected-chipmunk:~$ cat immutable
cat: immutable: Input/output error

It fails with EIO as mentioned here