edk2-firmware-tegra on wip-jetpack-5-ga with AGX Orin

[acostach]

Hi @madisongh,

This is more of a question then a bug report. I've built the edk2-firmware-tegra from the wip-jetpack-5-ga branch and it doesn't seem to work on my AGX Orin. Previously I've built and used the one from wip-jetpack-5-dp and it worked just fine.

With this one I'm getting:

bpmp: init complete
��size: 0x400000), name: B_secure-os
I> MB2-params @ 0x40060000
I> cpubl: Authentication Finalize Done
I> Binary cpubl loaded successfully at 0x8��DCE: FW Boot Done
��2c400000
I> tos: Authentication Finalize Done
I> Binary tos loaded successfully at 0x831e06000
I> Relocating OP-TEE dtb from: 0x831eff700 to 0x82b700000, size: 0x996
I> [0] START: 0x80000000, SIZE: 0x7ab600000
I> [1] START: 0x82c400000, SIZE: 0x400000
I> [2] START: 0x82d000000, SIZE: 0x800000
I> Setting NS memory ranges to OP-TEE dtb finished.
I> Scratch RSV109: 0x00000013    active chain: 1
I> Partition name: B_eks
I> Size of partition: 262144
I> Binary@ device:3/0 block-107264 (partition size: 0x40000), name: B_eks
I> eks: Authentication Finalize Done
I> Binary eks loaded successfully at 0x82d000400
I> EKB detected (length: 0x410) @ VA:0x82d000400
I> Task: Prepare TOS params (0x50013ab0)
I> Setting EKB blob info to OPTEE dtb finished.
I> Setting OPTEE arg3: 0x82b700000
I> Task: OEM SC7 context save (0x500146bc)
I> oem sc7 context saved
I> Task: Disable MSS perf stats (0x50020d18)
I> Task: Program display sticky bits (0x50020c94)
I> Task: Storage device deinit (0x50001ed8)
I> Task: SMMU external bypass disable (0x50011eb4)
I> Task: SMMU init (0x50011dd0)
I> Task: Program CBB PCIE AMAP regions (0x50014b08)
I> Task: Program GICv3 registers (0x50020db8)
I> Task: Audit firewall settings (0x5001e1c0)
I> Task: Bootchain failure check (0x500026a4)
I> Current Boot-Chain Slot: 1
I> BR-BCT Boot-Chain is 0, and status is 1. Set UPDATE_BRBCT bit to 1
I> MB2 finished

��NOTICE:  BL31: v2.6(release):4fa405dbd
NOTICE:  BL31: Built : 20:16:55, Aug 10 2022
I/TC: 
I/TC: Non-secure external DT found
I/TC: OP-TEE version: 3.16 (gcc version 9.3.0 (Buildroot 2020.08)) #2 Thu Aug 11 03:23:19 UTC 2022 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: WARNING: Test OEM keys are being used!
I/TC: This is only for TZ-SE testing and should NOT be used for a shipping product!
E/TC:00 0 jetson_user_key_pta_init:700 jetson_user_key_pta_init: Failed (f0100006).
E/TC:00 0 call_initcalls:43 Initcall __text_start + 0x000dab50 failed
I/TC: Primary CPU switching to normal world boot
��
  Jetson UEFI firmware (version v35.1.0 built on 2022-08-11T18:01:41+00:00)

... outputs tens of blank new-lines for a couple seconds, then...

Synchronous Exception at 0x0000000803065734
ASSERT [ArmCpuDxe] /home/acostach/work/xavier_nx/jetson-release/balena-jetson/build_orin2/tmp/work/jetson_agx_orin_devkit-poky-linux/edk2-firmware-tegra/35.1.0-r0/edk2-tegra/edk2/ArmPkg/Library/DefaultException)

Recursive exception occurred while dumping the CPU state

                                                        Recursive exception occurred while dumping the CPU state
                                                                                                                ASSERT [ArmCpuDxe] /home/acostach/work/xavier_nx/jetson-release/balena-jetson/build_orin2/tmp/work)

Resetting the system in 5 seconds.
����Shutdown state requested 1
Rebooting system ...

I also can't enter the uefi menu by pressing the Esc key like I could on 34.1.1, perhaps because of this crash.

My question is, does edk2-firmware-tegra work fine for you on the AGX Orin? If it does, then perhaps I'm doing something wrong in my setup, although I haven't patched this package yet.

Thank you

[dwalkes]

Cross linking test thread on this in https://gitter.im/OE4T/jetpack-5-test?at=63123cb6aa09177429ee85fe - Looks like @ichergui has reproduced.

[acostach]

Hi @ichergui, do you know if there's any workaround or potential fix for using edk2-firmware-tegra from sources on the AGX Orin?

Thanks

[ichergui]

Hey @acostach Sorry for the late reply. I will try to work on it over the weekend. I need to sync with @madisongh first I will keep you updated.

[acostach]

Thank you!

[acostach]

Hi @ichergui, just checking if you had time to look over this. Thank you!

[ichergui]

I'm sorry @acostach I don't have updates for now. I'm still investigating the issue :disappointed:

[ichergui]

Hey @acostach

As promised, I tried new fixes from nvidia on edk2-nvidia repo via branch rel-35-updates but still getting the same issue I need to investigate more. Thanks for @madisongh for his suggestion.

Attached file contains complete log from serial UART uefi_debug_mode_orin_agx_branch_rel-35-updates.log

[ichergui]

@acostach FYI, I create post in the Nvidia forum. Hopefully the nvidia team can help me to progress on this issue. Here is the link: https://forums.developer.nvidia.com/t/building-edk2-firmware-for-tegra-with-gcc-12-2/227757

[acostach]

thank you for the update @ichergui, hopefully Nvidia can tell what can be done to fix this

[mikhail-vl]

@ichergui @acostach Do you have any updates on this issue? NVidia recommended using their toolchain. It's not clear if it's possible for balena processes.

We can help with testing Orin build as soon as it's ready.

Thank you.

[ichergui]

Hey @mikhail-vl I'm sorry, I don't have updates. Regarding the toolchain, we are using the one from the OE-Core. I will let @madisongh share his thoughts concerning the right toolchain to use

[acostach]

Hi Mikhail, BalenaOS uses meta-tegra as the upstream BSP. Once the issue is fixed here and we can boot the device using the firmware built from sources, we can patch it for BalenaOS and continue to look into supporting the AGX Orin.

[mikhail-vl]

@acostach Thank you for the update. Do you have a timeline for this issue?

Anything we can do to help?

[acostach]

@mikhail-vl I cannot offer a timeline, perhaps @ichergui or @madisongh can tell.

[madisongh]

I finally got my hands on an Orin dev kit and was able to look into this. Turns out I missed updating the SRCREV of the repo that contains the NvGop pre-built driver (which is only used on Orin) when I originally updated the recipe for R35.1.0. The fix is in master and kirkstone now.