search

OE4T / meta-tegra

BSP layer for NVIDIA Jetson platforms, based on L4T
MIT License
397 stars 220 forks source link

TX2 bootloader update issue #744

Closed lfdmn closed 3 years ago

lfdmn commented 3 years ago

Hi,

I'm porting a TX2 image from warrior to dunfell-l4t-r32.4.3 after a recent shipment of D02 modules not compatible with prior jetpack revision.

The booloader update succeeds

nv_update_engine -i
Nvidia A/B-Redundancy Update tool Version 1.2
current slot 0
Start updating slot 1
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 131072
BLOB_SIZE 42891591
HEADER_SIZE 40
NUMBER_OF_ELEMENTS 49
HEADER_TYPE 0
UNCOMP_SIZE 42891591
MB1_RATCHET_LV 98
MTS_RATCHET_LV 112
ROLLBACK_FUSE_LV 109
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 3310-D01-1000-B.0-1-0-astro-tx2-mmcblk0p1
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 3310-D01-1000-B.0-1-0-astro-tx2-mmcblk0p1
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
bpmp-fw  5920  533904  2    1  1
sce-fw  539824  125568  2    1  1
cpu-bootloader  665392  285216  2    1  1
mb2  950608  117568  2    0  1
spe-fw  1068176  95984  2    0  1
secure-os  1164160  366400  2    1  1
adsp-fw  1530560  108208  2    1  1
BMP  1638768  75973  2    1  1
mb1  1714741  99664  2    2  1
mts-bootpack  1814405  2082144  2    1  1
mts-preboot  3896549  63104  2    0  1
sc7  3959653  23632  2    1  1
bpmp-fw-dtb  3983285  605120  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  4588405  605120  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  5193525  605120  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  5798645  605120  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  6403765  605120  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  1  1
bpmp-fw-dtb  7008885  605120  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  1  0
bootloader-dtb  7614005  240576  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  1  0
bootloader-dtb  7854581  240576  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  1  0
bootloader-dtb  8095157  240576  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  1  0
bootloader-dtb  8335733  240576  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  1  0
bootloader-dtb  8576309  240576  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  1  1
bootloader-dtb  8816885  240576  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  1  0
VER  9057461  102  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  0  0
VER  9057563  102  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  0  0
VER  9057665  102  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  0  0
VER  9057767  101  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  0  0
VER  9057868  102  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  0  1
VER  9057970  101  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  0  0
BCT  9058071  3584  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  2  0
BCT  9061655  3584  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  2  0
BCT  9065239  3584  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  2  0
BCT  9068823  3584  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  2  0
BCT  9072407  3584  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  2  1
BCT  9075991  3584  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  2  0
MB1_BCT  9079575  50096  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  0  0
MB1_BCT  9129671  50096  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  0  0
MB1_BCT  9179767  50096  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  0  0
MB1_BCT  9229863  50096  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  0  0
MB1_BCT  9279959  50096  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  0  1
MB1_BCT  9330055  50096  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  0  0
kernel  9380151  32067984  2    1  1
kernel-dtb  41448135  240576  2  3310-B00-1000--1-0-astro-tx2-mmcblk0p1  1  0
kernel-dtb  41688711  240576  2  3310-B02-1000--1-0-astro-tx2-mmcblk0p1  1  0
kernel-dtb  41929287  240576  2  3310-C04-1000--1-0-astro-tx2-mmcblk0p1  1  0
kernel-dtb  42169863  240576  2  3310-D00-1000--1-0-astro-tx2-mmcblk0p1  1  0
kernel-dtb  42410439  240576  2  3310-D01-1000--1-0-astro-tx2-mmcblk0p1  1  1
kernel-dtb  42651015  240576  2  3310-D02-1000--1-0-astro-tx2-mmcblk0p1  1  0
Saving Entry table to /opt/ota_package/entry_table
Writing 533904 bytes to /dev/disk/by-partlabel/bpmp-fw_b for bpmp-fw
Writing 125568 bytes to /dev/disk/by-partlabel/sce-fw_b for sce-fw
Writing 285216 bytes to /dev/disk/by-partlabel/cpu-bootloader_b for cpu-bootloader
mb2 write: slot = 1 offset = 1276928 bytes = 117568
spe-fw write: slot = 1 offset = 883712 bytes = 95984
Writing 366400 bytes to /dev/disk/by-partlabel/secure-os_b for secure-os
Writing 108208 bytes to /dev/disk/by-partlabel/adsp-fw_b for adsp-fw
Writing 75973 bytes to /dev/disk/by-partlabel/BMP_b for BMP
Writing 2082144 bytes to /dev/disk/by-partlabel/mts-bootpack_b for mts-bootpack
mts-preboot write: slot = 1 offset = 1801216 bytes = 63104
Writing 23632 bytes to /dev/disk/by-partlabel/sc7_b for sc7
Writing 605120 bytes to /dev/disk/by-partlabel/bpmp-fw-dtb_b for bpmp-fw-dtb
Writing 240576 bytes to /dev/disk/by-partlabel/bootloader-dtb_b for bootloader-dtb
VER write: slot = 1 offset = 8257536 bytes = 102
MB1_BCT write: slot = 1 offset = 622592 bytes = 50096
Writing 32067984 bytes to /dev/disk/by-partlabel/kernel_b for kernel
Writing 240576 bytes to /dev/disk/by-partlabel/kernel-dtb_b for kernel-dtb
mb1 write: slot = 0 offset = 32768 bytes = 99664
BCT slot = 0 write: offset = 3584 bytes = 3584
BCT slot = 0 write: offset = 0 bytes = 3584
Update Rollback partitions
Writing 533904 bytes to /dev/disk/by-partlabel/bpmp-fw for bpmp-fw
Writing 125568 bytes to /dev/disk/by-partlabel/sce-fw for sce-fw
Writing 285216 bytes to /dev/disk/by-partlabel/cpu-bootloader for cpu-bootloader
mb2 write: slot = 0 offset = 1014784 bytes = 117568
spe-fw write: slot = 0 offset = 752640 bytes = 95984
Writing 366400 bytes to /dev/disk/by-partlabel/secure-os for secure-os
Writing 108208 bytes to /dev/disk/by-partlabel/adsp-fw for adsp-fw
Writing 75973 bytes to /dev/disk/by-partlabel/BMP for BMP
Writing 2082144 bytes to /dev/disk/by-partlabel/mts-bootpack for mts-bootpack
mts-preboot write: slot = 0 offset = 1539072 bytes = 63104
Writing 23632 bytes to /dev/disk/by-partlabel/sc7 for sc7
Writing 605120 bytes to /dev/disk/by-partlabel/bpmp-fw-dtb for bpmp-fw-dtb
Writing 240576 bytes to /dev/disk/by-partlabel/bootloader-dtb for bootloader-dtb
VER write: slot = 0 offset = 8323072 bytes = 102
MB1_BCT write: slot = 0 offset = 557056 bytes = 50096
Writing 32067984 bytes to /dev/disk/by-partlabel/kernel for kernel
Writing 240576 bytes to /dev/disk/by-partlabel/kernel-dtb for kernel-dtb
Update successful
Setting slot 1 as active boot slot
Rebooting system....10.9.8.7.6.5.4.3.2.1

The board boots to the correct next slot. 

nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 0,             suffix: _a,             retry_count: 0,             boot_successful: 0
slot: 1,             priority: 15,             suffix: _b,             retry_count: 7,             boot_successful: 1

But the verification at the next boot fails:

nv_update_engine -v
Nvidia A/B-Redundancy Update tool Version 1.2
verifying update
The current slot 0 is marked as boot successful
SM: S22
Previous update on slot 1 failed

I've had a similar issue in the past. There I was missing the entry_table file in /opt/ota_package from APP to APP_b partition. This is not the case now and entry_table is copied and present.

If I delete entry_table from the new partition I get the same error so it seems unrelated.

I'm not getting much from strace

strace nv_update_engine -v
execve("/usr/sbin/nv_update_engine", ["nv_update_engine", "-v"], 0x7feffd0318 /* 16 vars */) = 0
brk(NULL)                               = 0x2dbeb000
brk(0x2dbebf60)                         = 0x2dbebf60
uname({sysname="Linux", nodename="localhost", ...}) = 0
readlinkat(AT_FDCWD, "/proc/self/exe", "/usr/sbin/nv_update_engine", 4096) = 26
brk(0x2dc0cf60)                         = 0x2dc0cf60
brk(0x2dc0d000)                         = 0x2dc0d000
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "Nvidia A/B-Redundancy Update too"..., 46Nvidia A/B-Redundancy Update tool Version 1.2
) = 46
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 1636, SEEK_SET)                = 1636
read(3, "\0\0\3\0\276\17\0\0\0\20\0\0", 12) = 12
close(3)                                = 0
write(1, "verifying update\n", 17verifying update
)      = 17
openat(AT_FDCWD, "/proc/cmdline", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "console=ttyS0,115200 console=tty"..., 1024) = 424
read(3, "", 1024)                       = 0
close(3)                                = 0
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
write(1, "The current slot 0 is marked as "..., 48The current slot 0 is marked as boot successful
) = 48
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
write(1, "SM: S22\n", 8SM: S22
)                = 8
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
openat(AT_FDCWD, "/etc/nv_boot_control.conf", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146, ...}) = 0
read(3, "TNSPEC 3310-D01-1000-B.0-1-0-ast"..., 1024) = 146
close(3)                                = 0
openat(AT_FDCWD, "/dev/mmcblk0boot0", O_RDWR) = 3
lseek(3, 2063360, SEEK_SET)             = 2063360
faccessat(AT_FDCWD, "/sys/block/mmcblk0boot0/force_ro", F_OK) = 0
openat(AT_FDCWD, "/sys/block/mmcblk0boot0/force_ro", O_RDWR) = 4
write(4, "0", 1)                        = 1
close(4)                                = 0
write(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
lseek(3, 2067456, SEEK_SET)             = 2067456
write(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
faccessat(AT_FDCWD, "/sys/block/mmcblk0boot0/force_ro", F_OK) = 0
openat(AT_FDCWD, "/sys/block/mmcblk0boot0/force_ro", O_RDWR) = 4
write(4, "1", 1)                        = 1
close(4)                                = 0
lseek(3, 2063360, SEEK_SET)             = 2063360
read(3, "\0NBC\3\3\2\0\17_a\7\1\0_b\0\0\246\337\326\262", 22) = 22
close(3)                                = 0
write(1, "Previous update on slot 1 failed"..., 33Previous update on slot 1 failed
) = 33
exit_group(0)                           = ?
+++ exited with 0 +++

The kernel cmd has the correct slot info ... boot.slot_suffix=_b ...

/etc/nv_boot_control.conf looks ok to me

cat /etc/nv_boot_control.conf 
TNSPEC 3310-D01-1000-B.0-1-0-astro-tx2-mmcblk0p1
TEGRA_CHIPID 0x18
TEGRA_OTA_BOOT_DEVICE /dev/mmcblk0boot0
TEGRA_OTA_GPT_DEVICE /dev/mmcblk0boot1

The bootloader update works with Xavier based machines on the same branch.

Any hint where to look next?

Thanks,

madisongh commented 3 years ago

What version of the BSP was on your A partition vs what was being installed on B? If they are different, then the next question:

Do you chroot into the newly-installed (B) partition to run nv-update-engine, or were using the one in the A partition? I know I've run into issues with cross-overs (trying to install a BUP from a later BSP using the tools/config from an older BSP version) not working, which is why I took the chroot path for the Mender (and other) integration. Doing that should also eliminate the need to copy the entry_table file over, too, since it should end up in the B partition's filesystem.

lfdmn commented 3 years ago

For this test I'm re-deploying the same bootloader payload. So the BSP is the same.

On slot A I nv_update_engine_install --install no-reboot, mount the B partition, copy the entry_table, reboot to slot B and run nv_update_engine_install --verify

The entry_table looks fine to me too:

bpmp-fw,5920,533904,2,,1,1
sce-fw,539824,125568,2,,1,1
cpu-bootloader,665392,285216,2,,1,1
mb2,950608,117568,2,,0,1
spe-fw,1068176,95984,2,,0,1
secure-os,1164160,366400,2,,1,1
adsp-fw,1530560,108208,2,,1,1
BMP,1638768,75973,2,,1,1
mb1,1714741,99664,2,,2,1
mts-bootpack,1814405,2082144,2,,1,1
mts-preboot,3896549,63104,2,,0,1
sc7,3959653,23632,2,,1,1
bpmp-fw-dtb,6403765,605120,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,1,1
bootloader-dtb,8576309,240576,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,1,1
VER,9057869,101,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,0,1
BCT,9072408,3584,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,2,1
MB1_BCT,9279960,50096,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,0,1
kernel,9380152,32067984,2,,1,1
kernel-dtb,42410440,240576,2,3310-D01-1000--1-0-astro-tx2-mmcblk0p1,1,1

chroot sounds like a good idea. I'll give this a try.

lfdmn commented 3 years ago

I'm actually getting the same problem with tegra-demo-distro.

I did the following:

git clone https://github.com/OE4T/tegra-demo-distro.git
cd tegra-demo-distro
git checkout dunfell-l4t-r32.4.3
git submodule update --init

Changed demo-image-base.bb as:

DESCRIPTION = "Tegra demo base image"

require demo-image-common.inc

IMAGE_INSTALL_append = "\
    tegra-redundant-boot \
    tegra-bup-payload \
"

Added PREFERRED_PROVIDER_virtual/bootloader_tegra186 = "cboot-t18x" to tegrademo.conf

Build and flash

. ./setup-env --machine jetson-tx2
bitbake `demo-image-base
cd tmp/deploy/images/jetson-tx2/
mkdir demo-image-base && cd demo-image-base
tar -xvzf ../demo-image-base-jetson-tx2.tegraflash.tar.gz
# boot TX2 devkit to flashing mode
sudo ./doflash.sh

After the first boot

$ nv_update_engine --enable-ab
$ nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 15,             suffix: _a,             retry_count: 7,             boot_successful: 1
slot: 1,             priority: 14,             suffix: _b,             retry_count: 7,             boot_successful: 1

$ nv_update_engine --install
Nvidia A/B-Redundancy Update tool Version 1.2
current slot 0
Start updating slot 1
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 131072
BLOB_SIZE 13008176
HEADER_SIZE 40
NUMBER_OF_ELEMENTS 49
HEADER_TYPE 0
UNCOMP_SIZE 13008176
MB1_RATCHET_LV 98
MTS_RATCHET_LV 112
ROLLBACK_FUSE_LV 109
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 3310-D01-1000-B.0-1-0-jetson-tx2-mmcblk0p1
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 3310-D01-1000-B.0-1-0-jetson-tx2-mmcblk0p1
Device is fused board.
ENTRY_TABLE:
PART  POS  LEN  VER TNSPEC TYPE UPDATABLE
bpmp-fw  5920  533904  2    1  1
sce-fw  539824  125568  2    1  1
cpu-bootloader  665392  287232  2    1  1
mb2  952624  117568  2    0  1
spe-fw  1070192  95984  2    0  1
secure-os  1166176  366400  2    1  1
adsp-fw  1532576  108208  2    1  1
BMP  1640784  70988  2    1  1
mb1  1711772  99664  2    2  1
mts-bootpack  1811436  2082144  2    1  1
mts-preboot  3893580  63104  2    0  1
sc7  3956684  23632  2    1  1
bpmp-fw-dtb  3980316  605120  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  4585436  605120  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  5190556  605120  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  5795676  605120  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bpmp-fw-dtb  6400796  605120  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  1  1
bpmp-fw-dtb  7005916  605120  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bootloader-dtb  7611036  371808  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bootloader-dtb  7982844  371808  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bootloader-dtb  8354652  371808  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bootloader-dtb  8726460  371808  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
bootloader-dtb  9098268  371808  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  1  1
bootloader-dtb  9470076  371808  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
VER  9841884  102  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  0  0
VER  9841986  102  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  0  0
VER  9842088  102  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  0  0
VER  9842190  102  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  0  0
VER  9842292  102  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  0  1
VER  9842394  102  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  0  0
BCT  9842496  3584  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  2  0
BCT  9846080  3584  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  2  0
BCT  9849664  3584  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  2  0
BCT  9853248  3584  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  2  0
BCT  9856832  3584  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  2  1
BCT  9860416  3584  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  2  0
MB1_BCT  9864000  50096  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  0  0
MB1_BCT  9914096  50096  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  0  0
MB1_BCT  9964192  50096  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  0  0
MB1_BCT  10014288  50096  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  0  0
MB1_BCT  10064384  50096  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  0  1
MB1_BCT  10114480  50096  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  0  0
kernel  10164576  612752  2    1  1
kernel-dtb  10777328  371808  2  3310-B00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
kernel-dtb  11149136  371808  2  3310-B02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
kernel-dtb  11520944  371808  2  3310-C04-1000--1-0-jetson-tx2-mmcblk0p1  1  0
kernel-dtb  11892752  371808  2  3310-D00-1000--1-0-jetson-tx2-mmcblk0p1  1  0
kernel-dtb  12264560  371808  2  3310-D01-1000--1-0-jetson-tx2-mmcblk0p1  1  1
kernel-dtb  12636368  371808  2  3310-D02-1000--1-0-jetson-tx2-mmcblk0p1  1  0
Saving Entry table to /opt/ota_package/entry_table
Writing 533904 bytes to /dev/disk/by-partlabel/bpmp-fw_b for bpmp-fw
Writing 125568 bytes to /dev/disk/by-partlabel/sce-fw_b for sce-fw
Writing 287232 bytes to /dev/disk/by-partlabel/cpu-bootloader_b for cpu-bootloader
mb2 write: slot = 1 offset = 1276928 bytes = 117568
spe-fw write: slot = 1 offset = 883712 bytes = 95984
Writing 366400 bytes to /dev/disk/by-partlabel/secure-os_b for secure-os
Writing 108208 bytes to /dev/disk/by-partlabel/adsp-fw_b for adsp-fw
Writing 70988 bytes to /dev/disk/by-partlabel/BMP_b for BMP
Writing 2082144 bytes to /dev/disk/by-partlabel/mts-bootpack_b for mts-bootpack
mts-preboot write: slot = 1 offset = 1801216 bytes = 63104
Writing 23632 bytes to /dev/disk/by-partlabel/sc7_b for sc7
Writing 605120 bytes to /dev/disk/by-partlabel/bpmp-fw-dtb_b for bpmp-fw-dtb
Writing 371808 bytes to /dev/disk/by-partlabel/bootloader-dtb_b for bootloader-dtb
VER write: slot = 1 offset = 8257536 bytes = 102
MB1_BCT write: slot = 1 offset = 622592 bytes = 50096
Writing 612752 bytes to /dev/disk/by-partlabel/kernel_b for kernel
Writing 371808 bytes to /dev/disk/by-partlabel/kernel-dtb_b for kernel-dtb
mb1 write: slot = 0 offset = 32768 bytes = 99664
BCT slot = 0 write: offset = 3584 bytes = 3584
BCT slot = 0 write: offset = 0 bytes = 3584
Update Rollback partitions
Writing 533904 bytes to /dev/disk/by-partlabel/bpmp-fw for bpmp-fw
Writing 125568 bytes to /dev/disk/by-partlabel/sce-fw for sce-fw
Writing 287232 bytes to /dev/disk/by-partlabel/cpu-bootloader for cpu-bootloader
mb2 write: slot = 0 offset = 1014784 bytes = 117568
spe-fw write: slot = 0 offset = 752640 bytes = 95984
Writing 366400 bytes to /dev/disk/by-partlabel/secure-os for secure-os
Writing 108208 bytes to /dev/disk/by-partlabel/adsp-fw for adsp-fw
Writing 70988 bytes to /dev/disk/by-partlabel/BMP for BMP
Writing 2082144 bytes to /dev/disk/by-partlabel/mts-bootpack for mts-bootpack
mts-preboot write: slot = 0 offset = 1539072 bytes = 63104
Writing 23632 bytes to /dev/disk/by-partlabel/sc7 for sc7
Writing 605120 bytes to /dev/disk/by-partlabel/bpmp-fw-dtb for bpmp-fw-dtb
Writing 371808 bytes to /dev/disk/by-partlabel/bootloader-dtb for bootloader-dtb
VER write: slot = 0 offset = 8323072 bytes = 102
MB1_BCT write: slot = 0 offset = 557056 bytes = 50096
Writing 612752 bytes to /dev/disk/by-partlabel/kernel for kernel
Writing 371808 bytes to /dev/disk/by-partlabel/kernel-dtb for kernel-dtb
Update successful
Setting slot 1 as active boot slot
Rebooting system....10.9.8.7.6.5.4.3.2.1

# after reboot
$ nvbootctrl dump-slots-info
magic:0x43424e00,             version: 3             features: 3             num_slots: 2
slot: 0,             priority: 0,             suffix: _a,             retry_count: 0,             boot_successful: 0
slot: 1,             priority: 15,             suffix: _b,             retry_count: 7,             boot_successful: 1

$ ls -la /opt/ota_package/
total 12664
drwxr-xr-x    2 root     root          4096 Sep 20 10:47 .
drwxr-xr-x    4 root     root          4096 Mar  9  2018 ..
-rw-r--r--    1 root     root      13008176 Mar  9  2018 bl_update_payload
-rw-r--r--    1 root     root           796 Sep 20 10:47 entry_table

$ cat /opt/ota_package/entry_table 
bpmp-fw,5920,533904,2,,1,1
sce-fw,539824,125568,2,,1,1
cpu-bootloader,665392,287232,2,,1,1
mb2,952624,117568,2,,0,1
spe-fw,1070192,95984,2,,0,1
secure-os,1166176,366400,2,,1,1
adsp-fw,1532576,108208,2,,1,1
BMP,1640784,70988,2,,1,1
mb1,1711772,99664,2,,2,1
mts-bootpack,1811436,2082144,2,,1,1
mts-preboot,3893580,63104,2,,0,1
sc7,3956684,23632,2,,1,1
bpmp-fw-dtb,6400796,605120,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,1,1
bootloader-dtb,9098268,371808,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,1,1
VER,9842292,102,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,0,1
BCT,9856832,3584,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,2,1
MB1_BCT,10064384,50096,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,0,1
kernel,10164576,612752,2,,1,1
kernel-dtb,12264560,371808,2,3310-D01-1000--1-0-jetson-tx2-mmcblk0p1,1,1

$ nv_update_engine --verify
Nvidia A/B-Redundancy Update tool Version 1.2
verifying update
The current slot 1 is marked as boot successful
SM: S22
Previous update on slot 0 failed

Any chance someone could try this?

Thanks

madisongh commented 3 years ago

I tested this with both the demo distro and with stock L4T R32.4.3 and yes, the verification fails on reboot. Looks like nv_update_engine (actually, nv_bootloader_payload_updater, which is called on by nv_update_engine) has a bug in it in R32.4.3 where it doesn't account for the fact that the partition offsets in the partition table at the end of /dev/mmcblk0boot1 are based on treating the two eMMC boot partitions as a single device. So when it seeks to the start of the VER or VER_b partition, which is located in the boot1 device, it gets an EINVAL error from the kernel (because it's seeking on the boot0 device to an offset greater than the size of that device), which it doesn't check before reading/writing. On the verification pass, it then reads whatever garbage happens to be at its current position (because the seek failed) so the comparision fails.

This appears to be fixed in R32.4.4, and it doesn't really affect stock L4T R32.4.3 because they do bootloader updates only to update to R32.4.4 - so after it reboots, it's running the fixed version of the tools and everything checks out OK.

We don't see this with the Mender-enabled demo distro because we disable the nv_update_verifier service - each rootfs is tightly coupled to the bootloader contents associated with the boot slot (since the bootloader contents has stuff like firmware, device tree, and maybe the kernel as well, depending on your setup), so with a full dual-rootfs A/B setup, you don't really want the verifier (which copies the current bootslot contents back to the other slot).

lfdmn commented 3 years ago

Thanks for the explanation!

The dunfell-l4t-r32.5.0 branch works for me for TX2. Checking with Xavier now.

I found nv_update_engine issues in gatesgarth branch. nv_update_engine -i fails saying eks partition isn't found. Looking at the system traces, it tries to mount eks_b partition which is missing from the stock flash layout.

I added it, the install went through successfully but the next boot hangs. I have not stored encryption keys myself. Out of curiousity, has there been a need for a _b partition for the keys?

madisongh commented 3 years ago

I found nv_update_engine issues in gatesgarth branch. nv_update_engine -i fails saying eks partition isn't found.

Yeah, I saw that, too, and pushed 9c0bf75a3a1827215ead20503b1e372137f48e3d to fix it.

Out of curiousity, has there been a need for a _b partition for the keys?

It provides a way to add/update/rotate keys to go with an updated TOS, if you need to.