OE4T / meta-tegra

BSP layer for NVIDIA Jetson platforms, based on L4T
MIT License
410 stars 225 forks source link

Mender update for the Jetson Xavier build fails #268

Closed focusware-develop closed 4 years ago

focusware-develop commented 4 years ago

Hi Matt,

I'm testing the test-distro build for the zeus-mender-l4t-r32.3.1 branch on a Jetson Xavier. When running the mender installer I get the following error:

mender -install core-image-minimal-jetson-xavier.mender

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf module=config INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config INFO[0000] Mender running on partition: /dev/mmcblk0p1 module=main INFO[0000] Start updating from local image file: [core-image-minimal-jetson-xavier.mender] module=standalone Installing Artifact of size 65272320... INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available module=modules INFO[0000] installer: authenticated digital signature of artifact module=installer INFO[0000] native sector size of block device /dev/mmcblk0p42 is 512, we will write in chunks of 1048576 module=dual_rootfs_device INFO[0000] opening device /dev/mmcblk0p42 for writing module=block_device INFO[0000] partition /dev/mmcblk0p42 size: 15034482688 module=block_device ................................ 1% 1024 KiB ................................ 3% 2048 KiB ................................ 4% 3072 KiB ................................ 6% 4096 KiB ... ................................ 93% 59392 KiB ................................ 94% 60416 KiB ................................ 96% 61440 KiB ................................ 97% 62464 KiB ................................ 99% 63488 KiB .......INFO[0145] wrote 14282760192/14282760192 bytes of update to device /dev/mmcblk0p42 module=dual_rootfs_device 100% 63742 KiB INFO[0147] Enabling partition with new image installed to be a boot candidate: 42 module=dual_rootfs_device ERRO[0147] stderr collected while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update [ERR: bootloader update failed ] module=executor ERRO[0147] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 module=standalone Rolling back Artifact... INFO[0147] setting partition for rollback: 42 module=dual_rootfs_device ERRO[0147] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 module=main

When I start the ArtifactInstall_Leave_80_bl-update script from the command line I'll get the following error:

/var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update

Installing NVIDIA bootloader update payload Current boot slot: 1 Nvidia A/B-Redundancy Update tool Version 1.2 current slot 1 Start updating slot 0 Tegra User Block Device: /dev/disk/by-partlabel Tegra Boot Block Device: /dev/mmcblk0boot0 HEADER: MAGIC NVIDIABLOBV2 HEX_VALUE 131072 BLOB_SIZE 46601064 HEADER_SIZE 48 NUMBER_OF_ELEMENTS 30 HEADER_TYPE 0 UNCOMP_SIZE 46601064 MB1_RATCHET_LV 0 MTS_RATCHET_LV 0 ROLLBACK_FUSE_LV 0 Device TN Spec: 2888-400-0001-L.0-1-2-jetson-xavier-mmcblk0p1 OTA Blob update failed. Status: 3 /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed. Slot 0 update failed ERR: bootloader update failed

Running the commands in the script by hand:

fw_printenv -n mender_boot_part 1

mktemp -d -t nvbup.XXXXXX /tmp/nvbup.SS7s6G

mount /dev/mmcblk0p1 /tmp/nvbup.SS7s6G mount --bind /sys "/tmp/nvbup.SS7s6G/sys" mount --bind /proc "/tmp/nvbup.SS7s6G/proc" mount --bind /dev "/tmp/nvbup.SS7s6G/dev"

chroot /tmp/nvbup.SS7s6G /usr/sbin/nv_update_engine --install no-reboot Nvidia A/B-Redundancy Update tool Version 1.2 current slot 1 Start updating slot 0 Tegra User Block Device: /dev/disk/by-partlabel Tegra Boot Block Device: /dev/mmcblk0boot0 HEADER: MAGIC NVIDIABLOBV2 HEX_VALUE 131072 BLOB_SIZE 46601064 HEADER_SIZE 48 NUMBER_OF_ELEMENTS 30 HEADER_TYPE 0 UNCOMP_SIZE 46601064 MB1_RATCHET_LV 0 MTS_RATCHET_LV 0 ROLLBACK_FUSE_LV 0 Device TN Spec: 2888-400-0001-L.0-1-2-jetson-xavier-mmcblk0p1 OTA Blob update failed. Status: 3 /usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed. Slot 0 update failed

cat etc/nv_boot_control.template TNSPEC @TNSPEC@ TEGRA_CHIPID 0x19 TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0 TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

Do you have any suggestions on how to proceed with this?

Best regards, Boris.

madisongh commented 4 years ago

Try using core-image-base instead of core-image-minimal for your testing. I'm don't think core-image-minimal will pull in everything that you need. I just ran a quick test using core-image-base on my Xavier, and the mender update worked fine.

focusware-develop commented 4 years ago

These are the steps I use to build the image

git clone https://github.com/madisongh/test-distro.git cd test-distro git checkout zeus-mender-l4t-r32.3.1 git submodule update --init --recursive source ./setup-env bitbake core-image-base

Next I've flashed the image to the Xavier development module using the core-image-base-jetson-xavier.tegraflash.zip file.

Then I reboot the Xavier and update the firmware:

mender -install core-image-base-jetson-xavier.mender

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf module=config INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config INFO[0000] Mender running on partition: /dev/mmcblk0p1 module=main INFO[0000] Start updating from local image file: [core-image-base-jetson-xavier.mender] module=standalone Installing Artifact of size 123669504... INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available module=modules INFO[0000] installer: authenticated digital signature of artifact module=installer INFO[0000] native sector size of block device /dev/mmcblk0p42 is 512, we will write in chunks of 1048576 module=dual_rootfs_device INFO[0000] opening device /dev/mmcblk0p42 for writing module=block_device INFO[0000] partition /dev/mmcblk0p42 size: 15034482688 module=block_device ................................ 0% 1024 KiB ................................ 1% 2048 KiB ................................ 2% 3072 KiB ................................ 3% 4096 KiB ... ................................ 98% 118784 KiB ................................ 99% 119808 KiB ..............................INFO[0178] wrote 14282760192/14282760192 bytes of update to device /dev/mmcblk0p42 module=dual_rootfs_device 100% 120771 KiB INFO[0180] Enabling partition with new image installed to be a boot candidate: 42 module=dual_rootfs_device ERRO[0181] stderr collected while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update [ERR: bootloader update failed ] module=executor ERRO[0181] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 module=standalone Rolling back Artifact... INFO[0181] setting partition for rollback: 42 module=dual_rootfs_device ERRO[0181] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 module=main

The same error occurs.

I also use the madisongh/digsigserver to sign the build artifact. The files I use to setup the signing server are:

sudo mkdir -p /opt/nvidia/L4T-32.3.1-tegra186 sudo tar -C /opt/nvidia/L4T-32.3.1-tegra186/ -xvf Tegra186_Linux_R32.3.1_aarch64.tbz2 sudo tar -C /opt/nvidia/L4T-32.3.1-tegra186/ -xvf secureboot_R32.3.1_aarch64.tbz2

madisongh commented 4 years ago

There are a couple of possibilities then.

First, I see your Xavier has a BOARDREV of "L.0". The BUP generator doesn't handle that board revision yet, so nv_update_engine may be complaining because it's not finding some needed boot files that are matched by TNSPEC. You could try adding

TEGRA_BUPGEN_SPECS_append = " fab=400;boardsku=0001;boardrev=L.0"

to your local.conf to see if that helps.

Second, if your device has the secure boot fuses set, then you'll need to make some modifications to digsigserver to have it sign the boot files properly. Right now, digsigserver only supports signing TX2 boot files, I haven't had the guts to risk bricking my one Xavier dev kit to get the secure boot support straightened out on that platform.

focusware-develop commented 4 years ago

Adding the "L.0" board revision the the local.conf file worked like a charm!

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf module=config INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config INFO[0000] Mender running on partition: /dev/mmcblk0p1 module=main INFO[0000] Start updating from local image file: [core-image-minimal-jetson-xavier.mender] module=standalone Installing Artifact of size 65401344... INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available module=modules INFO[0000] installer: authenticated digital signature of artifact module=installer INFO[0000] native sector size of block device /dev/mmcblk0p42 is 512, we will write in chunks of 1048576 module=dual_rootfs_device INFO[0000] opening device /dev/mmcblk0p42 for writing module=block_device INFO[0000] partition /dev/mmcblk0p42 size: 15034482688 module=block_device ................................ 1% 1024 KiB ................................ 3% 2048 KiB ................................ 4% 3072 KiB ... ................................ 97% 62464 KiB ................................ 99% 63488 KiB ...........INFO[0144] wrote 14282760192/14282760192 bytes of update to device /dev/mmcblk0p42 module=dual_rootfs_device 100% 63868 KiB INFO[0146] Enabling partition with new image installed to be a boot candidate: 42 module=dual_rootfs_device Use -commit to update, or -rollback to roll back the update. At least one payload requested a reboot of the device it updated.

Thank you for your assistance and keep up the good work! Boris.

I didn't dare to use the fuses either :-) I've only used the signing of the build artifacts and the bootloader.

I'll submit two other pull requests that fixes some other small problems in the firmware image.