Mender Image Upgrade failed - can't update bootloader

[Buanderie]

Same as #190 maybe. Not sure. I've built an image with mender support for my Jetson Nano EMMC, and can't get to install the mender artifact (performed as a test for future OTA updates).

root@jetcam:~# mender install jetcam-image-minimal-jetson-nano-devkit-emmc.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [jetcam-image-minimal-jetson-nano-devkit-emmc.mender] 
Installing Artifact of size 89011200...
INFO[0000] No public key was provided for authenticating the artifact 
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available 
INFO[0000] Opening device "/dev/mmcblk0p18" for writing 
INFO[0000] Native sector size of block device /dev/mmcblk0p18 is 512 bytes. Mender will write in chunks of 1048576 bytes 
.............................................................. - 100 %
INFO[0041] All bytes were successfully written to the new partition 
INFO[0041] The optimized block-device writer wrote a total of 4743 frames, where 0 frames did need to be rewritten (i.e., skipped) 
INFO[0042] Wrote 4972347392/4972347392 bytes to the inactive partition 
INFO[0042] Enabling partition with new image installed to be a boot candidate: 18 
INFO[0042] Executing script: ArtifactInstall_Leave_80_bl-update 
INFO[0042] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
ERR: cannot perform bootloader update

---------- end of script output 
ERRO[0042] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 
Rolling back Artifact...
INFO[0042] Rolling back to the active partition: (1).   
ERRO[0042] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1

So, I tried to run the ArtifactInstall_Leave_80_bl-update separately to investigate the matter:

root@jetcam:~# /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update 
Installing NVIDIA bootloader update payload
mount: /tmp/nvbup.r77OHf: /dev/mmcblk0p1 already mounted on /.
ERR: Missing /opt/ota_package directory in installed rootfs

Weird. Since the /opt/ota_package/bl_update_payload is there :'(

root@jetcam:~# ls /opt/ota_package/bl_update_payload -alh
-rw-r--r--    1 root     root        5.6M Mar  9  2018 /opt/ota_package/bl_update_payload
root@jetcam:~#

Rerunning the script a second time gives:

Installing NVIDIA bootloader update payload
mount: /tmp/nvbup.GpF3jc: /dev/mmcblk0p1 already mounted on /.
ERR: Missing /opt/ota_package directory in installed rootfs

Not sure that this is the actual partition that should be used for BL update

I have no idea what to do next, at this point... I'll keep investigating though. Any idea ?

Thanks

[dwalkes]

@Buanderie which branch are you using? I can try to reproduce.

[dwalkes]

I think this is the actual error

INFO[0042] Executing script: ArtifactInstall_Leave_80_bl-update 
INFO[0042] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
ERR: cannot perform bootloader update

So, I tried to run the ArtifactInstall_Leave_80_bl-update separately to investigate the matter:

mount: /tmp/nvbup.r77OHf: /dev/mmcblk0p1 already mounted on /.

I think this won't work unless it runs as a part of the update - since the chroot at https://github.com/mendersoftware/meta-mender-community/blob/a8247d94d660ea2322293eff44bd9c343a4b05fd/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script-uboot#L37 is picking the wrong variable for ${new_boot_part}. You'll need to modify that variable yourself if you want to try running outside mender update.

Weird. Since the /opt/ota_package/bl_update_payload is there :'(

It's running from the chroot, see https://github.com/mendersoftware/meta-mender-community/blob/a8247d94d660ea2322293eff44bd9c343a4b05fd/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script-uboot#L59, so the error message above is probably related and you aren't getting a root mount with your alternate partition as needed in order to perform the update with your test of the ArtifactInstall_Leave_80_bl-update script.

[Buanderie]

@Buanderie which branch are you using? I can try to reproduce.

I'm using the dunfell branch

[Buanderie]

I think this is the actual error

INFO[0042] Executing script: ArtifactInstall_Leave_80_bl-update 
INFO[0042] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
ERR: cannot perform bootloader update

So, I tried to run the ArtifactInstall_Leave_80_bl-update separately to investigate the matter:

mount: /tmp/nvbup.r77OHf: /dev/mmcblk0p1 already mounted on /.

I think this won't work unless it runs as a part of the update - since the chroot at https://github.com/mendersoftware/meta-mender-community/blob/a8247d94d660ea2322293eff44bd9c343a4b05fd/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script-uboot#L37 is picking the wrong variable for ${new_boot_part}. You'll need to modify that variable yourself if you want to try running outside mender update.

Weird. Since the /opt/ota_package/bl_update_payload is there :'(

It's running from the chroot, see https://github.com/mendersoftware/meta-mender-community/blob/a8247d94d660ea2322293eff44bd9c343a4b05fd/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script-uboot#L59, so the error message above is probably related and you aren't getting a root mount with your alternate partition as needed in order to perform the update with your test of the ArtifactInstall_Leave_80_bl-update script.

I can try and produce an artifact with a modified version of the script with some debug output, though

[dwalkes]

I've verified no issues with jetson-nano-devkit-emmc on dunfell with demo-image-base at https://github.com/OE4T/tegra-demo-distro/commit/d2fe4294a55297b3536612e8e1bcd32d3cae7721

root@jetson-nano-devkit-emmc:~# mender install /tmp/demo-image-base-jetson-nano-devkit-emmc.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/mmcblk0p1
INFO[0000] Start updating from local image file: [/tmp/demo-image-base-jetson-nano-devkit-emmc.mender]
Installing Artifact of size 166818304...
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
INFO[0000] Opening device "/dev/mmcblk0p18" for writing
INFO[0000] Native sector size of block device /dev/mmcblk0p18 is 512 bytes. Mender will write in chunks of 1048576 bytes
.............................................................. - 100 %
INFO[0030] All bytes were successfully written to the new partition
INFO[0030] The optimized block-device writer wrote a total of 4743 frames, where 0 frames did need to be rewritten (i.e., skipped)
INFO[0031] Wrote 4972347392/4972347392 bytes to the inactive partition
INFO[0031] Enabling partition with new image installed to be a boot candidate: 18
INFO[0031] Executing script: ArtifactInstall_Leave_80_bl-update
Use -commit to update, or -rollback to roll back the update.
At least one payload requested a reboot of the device it updated.
root@jetson-nano-devkit-emmc:~#

You might try

tegra-bootloader-update --dry-run /opt/ota_package/bl_update_payload

and see if it complains about a TNSPEC with your setup similar to https://github.com/OE4T/tegra-demo-distro/issues/190

[Buanderie]

Okay let's see...

root@jetcam:~# tegra-bootloader-update --dry-run /opt/ota_package/bl_update_payload
/opt/ota_package/bl_update_payload: bad header version
/opt/ota_package/bl_update_payload: No such file or directory

No idea what that should mean though :smile:

[Buanderie]

Okay let's see...

root@jetcam:~# tegra-bootloader-update --dry-run /opt/ota_package/bl_update_payload
/opt/ota_package/bl_update_payload: bad header version
/opt/ota_package/bl_update_payload: No such file or directory

No idea what that should mean though smile

Okay... This error message didn't even exist in the source code of the latest version of tegra-boot-tools . Weird... Until I figured I had an old recipe of tegra-boot-tools building an old version of it (2.3.0 - which was not compatible with my BUP) in my own layer (which has highest priority).

My bad.

Mender install is working, now :)

Thanks for you help, though !