generating and then placing .sig files in /boot for Xavier NX

[arrow53]

I have cboot set up to change my device tree based on different hardware configurations.

As part of this I place the different .dtb files in /boot and install an extlinux.conf file for the different options.

Right now I'm using commands such as

./l4t_sign_image.sh --file tegra194-p3668-all-p3509-0000.dtb --chip 0x19

to generate the corresponding .sig files I need and I manually copy them over.

I'd like to do this in a recipe but I'm unsure where to start. Does anyone know what recipe has access to the dtb files and also the scripts that NVIDIA ships with L4T?

[madisongh]

Looking at what that script does, you should be able to add a bbappend to the kernel recipe that does something like this:

do_install[depends] += "tegra186-flashtools-native:do_populate_sysroot"

sign_dtbfile() {
    local pathsave="$PATH"
    local dtbfile="$1"
    local sigfile="${1}.sig"
    local signedfile="$(basename $1 .dtb)_sigheader.dtb.encrypt"
    PATH="${STAGING_BINDIR_NATIVE}/tegra186-flash:$PATH"
    python3 ${STAGING_BINDIR_NATIVE}/tegra186-flash/tegraflash.py --chip 0x19 --key None --cmd "sign $1"
    dd if="$signedfile" of="$sigfile" bs=4096 count=1 status=none
    rm -f "$signedfile"
    PATH="$pathsave"
}

do_install_append() {
    rm -rf ${WORKDIR}/dtbsign
    mkdir -p ${WORKDIR}/dtbsign
    install -d ${D}/${KERNEL_IMAGEDEST}
    oldwd="$PWD"
    cd ${WORKDIR}/dtbsign
    for dtbfile in ${KERNEL_DEVICETREE}; do
        cp ${B}/arch/${ARCH}/boot/dts/$dtbfile .
    sign_dtbfile $(basename $dtbfile)
    install -m 0644 $(basename $dtbfile).sig ${D}/${KERNEL_IMAGEDEST}/
    done
    cd "$oldwd"
}

If you've got all of your variant device tree files enumerated in the KERNEL_DEVICETREE variable, that should generate and install the .sig file for each in /boot. None is used as the key - if you need to sign them for an actual secured device, you'd need to pass in the right RSA private key file instead.

[arrow53]

wow @madisongh you saved me a ton of work

I just had to add this and worked perfectly

FILES_${KERNEL_PACKAGE_NAME}-devicetree += "/${KERNEL_IMAGEDEST}/*.dtb.sig"

[arrow53]

@madisongh sorry to bug you but wondering if you have any insight into a problem I'm having.

First note is that I had to add this line to get the dtb to work

    dd if="$signedfile" of="$dtbfile" bs=4096 skip=1 status=none

I need to sign the kernel image too and I'm using basically the same steps as you have above for the dtb. But, I get this on boot:

[0004.532] I> Loading kernel sig file from rootfs ...
[0004.532] I> rootfs path: /sd/boot/Image.sig
[0004.545] I> Loading kernel binary from rootfs ...
[0004.545] I> rootfs path: /sd/boot/Image
[0007.458] I> Validate kernel ...
[0007.459] I> T19x: Authenticate kernel (bin_type: 37), max size 0x5000000
[0007.799] E> digest on binary did not match!!
[0007.799] C> OEM authentication of kernel payload failed!
[0007.799] W> Failed to validate kernel binary (err=1077936152, fail=0)

I think it's something about how they are packaged/deployed that cboot is getting angry about but I spent a day on it and am at a loss. If I scp the files off then put them back on it works fine.

[madisongh]

Interesting that you are having to extract the DTB from the signed file - the extracted contents should be identical to the original, since the signature is just prepended.

The same approach should work for the kernel image, if I'm reading the cboot sources right. From the errors reported, it sounds like there is a mismatch between the signature in the .sig file and the contents of Image. Is /boot/Image a regular file, or a symlink? I'm not sure cboot's file lookup code necessarily handles symlinks, so that's something to check.

[arrow53]

thanks. I'm in the cboot code and the hash is definitely different from the header. I'll keep digging.

I'm not sure what to make of the dd'ing the original file but I noticed that l4t_sign_image.sh does it is where I got the idea.

Once i figure this out I'll report back.

[arrow53]

I'm gonna reopen this as I think it's an issue on the yocto side. Plus the behavior is so weird.

• install of the Image file in boot results in a file with the wrong sha256sum and also won't boot

• I have cboot set up to I can ignore extlinux. If I do that and boot then look at the sha256sum of the /boot/Image file it is fine

• If I cp/mv the file around on the target then reboot same problem. cboot gives the wrong sha256sum

• If I scp the file to my host Ubuntu box then scp it back then reboot the sha256sum is fine and it boots fine

In the linux-tegra recipe is there something about how the file is populated that would cause the file to be manipulated?

I've also looked at all the copies of Image in build/tmp/work/jetson_xavier_nx_devkit-oe4t-linux/linux-tegra/4.9.140+gitAUTOINC+a58470bb0f-r0/ and they all have the correct sha256sum

[madisongh]

In the linux-tegra recipe is there something about how the file is populated that would cause the file to be manipulated?

Well, there's the initramfs bundling, which is enabled by default on recent branches. That shouldn't affect the image installed in /boot, though... if I'm reading kernel.bbclass right, the non-bundled image is saved and restored as part of the bundling process.

You might try deferring the generation of the signature file for the kernel image(s) to a rootfs postprocessing command, instead of doing it in the kernel recipe, so you can be sure that the file you're processing is the one that's actually installed in the rootfs.

[arrow53]

@madisongh thanks. Yes, I have a recipe where I literally just put files in place that I know work. That doesn't work either.

There must be something about either how yocto is packaging the files or something about how cboot is reading them.

I've seen a few options for dealing with firmware files to avoid binaries being manipulated. It must be something like this, but these options don't help

INHIBIT_PACKAGE_STRIP = "1"
INHIBIT_SYSROOT_STRIP = "1"
INHIBIT_PACKAGE_DEBUG_SPLIT  = "1"

[madisongh]

Well, good luck. I tried using the extlinux.conf file to see if I could reproduce your problem and ran into lovely data abort errors in cboot because it didn't contain exactly the right contents that cboot was expecting. Once I got past that, I could see the signature check failures being reported, although they were ignored. (In some cases, it looked to me like cboot was corrupting the file system, too.) All my file comparisons checked out, though - contents appeared to be identical on the SDcard and in my build tree.

It's possible that cboot's ext4 filesystem code isn't quite up to snuff, too - it might not support some of the ext4 features that get used by default in Yocto builds, but probably aren't being used in the older Ubuntu code base that NVIDIA develops on. The whole feature looks like a tacked-on afterthought (which it is, really) that probably isn't fully baked (which is likely). Sorry I can't be more help.

[arrow53]

Sorry I can't be more help.

Actually I think your tip about the rootfs postproceessing is a great help. I'm flailing but I see it being used in raspberry pi for firmware so I have an example to work off. I do think the true culprit is cboot/ext4 as you mentioned. The fact I can scp the file off then back on and that fixes it is bizarre.

[kekiefer]

Oh! That just reminded me -- I ran into this. The issue is indeed a deficiency in the cboot ext4 implementation that it doesn't handle sparse files that are created by e2fsprogs. When you're copying the file off, and putting it back, you're making the file "un-sparse", which is why it then works.

This was pretty awkward to work around in Yocto because of the way e2fsprogs works, and I ended up abandoning extlinux because of it. There's no way to disable the sparse copy that e2fsprogs uses when making file system images based on a directory besides patching it out, but this patch to e2fsprogs did work, if you don't mind carrying this baggage with you:

From 3ba21919f4fca47a168058e98bc25f220439806e Mon Sep 17 00:00:00 2001
From: Kurt Kiefer <kekiefer@gmail.com>
Date: Sat, 4 Jul 2020 17:01:54 -0700
Subject: [PATCH] Disable sparse copy

---
 misc/create_inode.c | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/misc/create_inode.c b/misc/create_inode.c
index e8d1df6b..741f7079 100644
--- a/misc/create_inode.c
+++ b/misc/create_inode.c
@@ -437,10 +437,6 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
            blen = fs->blocksize;
            if (blen > got - bpos)
                blen = got - bpos;
-           if (memcmp(ptr, zerobuf, blen) == 0) {
-               ptr += blen;
-               continue;
-           }
            err = ext2fs_file_llseek(e2_file, off + bpos,
                         EXT2_SEEK_SET, NULL);
            if (err)
@@ -578,18 +574,6 @@ static errcode_t copy_file(ext2_filsys fs, int fd, struct stat *statbuf,
    if (err)
        goto out;

-#if defined(SEEK_DATA) && defined(SEEK_HOLE)
-   err = try_lseek_copy(fs, fd, statbuf, e2_file, buf, zerobuf);
-   if (err != EXT2_ET_UNIMPLEMENTED)
-       goto out;
-#endif
-
-#if defined(FS_IOC_FIEMAP)
-   err = try_fiemap_copy(fs, fd, e2_file, buf, zerobuf);
-   if (err != EXT2_ET_UNIMPLEMENTED)
-       goto out;
-#endif
-
    err = copy_file_chunk(fs, fd, e2_file, 0, statbuf->st_size, buf,
                  zerobuf);
 out:
-- 
2.26.2

[arrow53]

@kekiefer what a relief to know I'm not insane.

I tried making these changes to e2fsprogs but it still seems to place the files into the root filesystem as sparse files. I can tell because I can sum the raw bytes when cboot loads the file and they aren't the same. I'm using tegra-demo-distro and meta-mender has a patch for e2fsprogs so maybe that is conflicting?

diff --git a/misc/mke2fs.conf.in b/misc/mke2fs.conf.in
index 00afd91..695dc3f 100644
--- a/misc/mke2fs.conf.in
+++ b/misc/mke2fs.conf.in
@@ -11,7 +11,7 @@
        features = has_journal
    }
    ext4 = {
-       features = has_journal,extent,huge_file,flex_bg,metadata_csum,64bit,dir_nlink,extra_isize
+       features = has_journal,extent,huge_file,flex_bg,64bit,dir_nlink,extra_isize
        inode_size = 256
    }
    small = {
-- 

[kekiefer]

I would do a bitbake -e e2fsprogs-native and verify that the patch is being applied (look for SRC_URI=). After that, make sure it really regenerates the file system images -- it seems very possible sstate changes to e2fsprogs-native wouldn't trigger this. You can likely save some time mounting the ext4 images with loopback rather than trying to flash again.

[arrow53]

well, fair amount of weirdness I can't explain. Applying the patch and using loopback to mount the sum and sha256 of the kernel look fine in the ext4 file. But, when I flash the image neither the sum or the sha256 of the image match. It could be when cboot mounts the filesystem it does something goofy. The patch seems to really upset systemd.

My plan right now is use cboot to check the file. If it doesn't match set a flag, enter linux, scp the file, then reboot. Pretty wonky but I at least know that works.

Thanks @kekiefer and @madisongh for helping me understand the problem.

[arrow53]

@madisongh sorry to bug you again on this. I was using this in dunfell

FILES_${KERNEL_PACKAGE_NAME}-devicetree += "/${KERNEL_IMAGEDEST}/*.dtb.sig"

but it seems to no longer install the files in boot in gatesgarth. Generally, something has changed as I see no files installed in boot anymore.

Is there an easy way to get this placement back? I looked at the kernel recipes between the layers and don't see a reason.

[madisongh]

@arrow53 Make sure you're including the kernel-image and kernel-devicetree packages in your image. In the dunfell branch, those were listed in the MACHINE_ESSENTIAL_EXTRA_RDEPENDS for the tegra194 machines. In the normal case, though, they aren't required, since they're flashed into separate partitions. I removed them in commit 10f92ff003812ec2f2f80523823530de3b78809f, so they don't show up in the rootfs by default any more.

[arrow53]

@madisongh you're so awesome.