Interacting With OP-TEE OS from a Ubuntu REE.

[ThePerfectComputer]

Hello. I'm trying to use Ubuntu 22.04 for ARM on Linux with OPTEE. I built TFA, OPTEE-OS, OPTEE-client, OPTEE-examples, and OPTEE-test from source. Then I built U-Boot, linking against TFA and OPTEE-OS. I also built the Linux kernel with OPTEE enabled as well as a debian rootfs.

Putting it all together, I booted into OPTEE enabled Ubuntu on the RockPi4C+. Unfortunately, this image I created is/was not stable. From time to time the kernel would panic. Furthermore, commands such as xtest or optee_examples_hello_world would fail with errors such as "Failed to get monotonic counter for REE FS, using 0".

Any suggestions how one might go about building out Ubuntu on OP-TEE?

[ivila]

I think the "Failed to get monotonic counter for REE FS, using 0" is just a warning, but not error:

I/TC: WARNING (insecure configuration): Failed to get monotonic counter for REE FS, using 0

I met the kernel panic while I am using raspberry pi 3b, some firmwares do not consider the TEE/REE transtering, resulting a firmware transaction timeout, and just die directly(which is very stupid)

So if you encounter a kernel panic problem, what I could suggest is using a stable kernel release. I have a rockpi 4b which is building with debian bookworm, and my linux kernel is 6.1.0, which runs well for the pass few months. I think the rockpi 4c+ should not be too different?

[ivila]

Just for reference.

name	repo	version
optee_os	https://github.com/OP-TEE/optee_os	3.20.0
build	https://github.com/OP-TEE/build	master
trusted-firmware-a	https://github.com/ARM-software/arm-trusted-firmware	v2.10.0
u-boot	https://github.com/u-boot/u-boot	v2022.10
linux	https://github.com/torvalds/linux	v6.1
debian	-	bookworm

[ThePerfectComputer]

Would you perhaps happen to be able to share the build scripts you used to successfully build the debian image?

[ivila]

I build my rockpi4b image by follow steps, you can try it your own, but do notice that something is different with devices(like the mmcblk1/mmcblk0, and the dtb files)

1) Linux Kernel

1. clone the mainline kernel
2. change arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4.dtsi, just like this commit
3. remember the CROSS_COMPILE variable when building optee-os

then run ARCH=arm64 CROSS_COMPILE={CROSS_COMPILE} make defconfig, checks the .config file for the following config:

CONFIG_TEE=y
CONFIG_OPTEE=y

and then ARCH=arm64 CROSS_COMPILE={CROSS_COMPILE} make bindeb-pkg -j {nproc} here we get 4 files: 1) linux-image-${version}.deb 2) linux-headers-${version}.deb 3) linux-libc-${version}.deb 4) arch/arm64/boot/dts/rockchip/rk3399-rock-pi-4b.dtb (you should use another dtb file as you are RockPi 4C+)

2) Debian Rootfs

use debootstrap to build it. run sudo apt install -y debootstrap to install it first. then run the following scripts:

# create a folder to place the files
mkdir rootfs

# initial, you can switch to another mirror, just replace the "http://mirrors.ustc.edu.cn/debian"
sudo debootstrap --foreign --verbose --arch=arm64 bookworm rootfs http://mirrors.ustc.edu.cn/debian

# copy the linux built files into it, will use them later
mkdir -p rootfs/tmp_install
cp ${PATH_TO_YOUR_FOUR_LINUX_FILES} rootfs/tmp_install

# chrootfs to next build
sudo LC_ALL=C LANGUAGE=C LANG=C chroot rootfs bash

after chroot, do the following:

# second stage
/debootstrap/debootstrap --second-stage --verbose

# setup host name, you can change it as you want
export HOSTNAME="rockpi4b"
echo $HOSTNAME > /etc/hostname
sed -i '/localhost/s/$/\t'"$HOSTNAME"'/g' /etc/hosts

# set timezone
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

# set root password
passwd root

# install some other tools, you can do other modifications here, for example, setting up wifi configuration, create new user.
sudo apt update
sudo apt install -y wpasupplicant  net-tools udhcpc parted u-boot-tools initramfs-tools vim sudo

# install linux kernel files, only after you install initramfs-tool, or you have to update-initramfs yourself
dpkg -i /tmp_install/*.deb

# copy dtb file
mv /tmp_install/*.dtb /boot

# change linux kernel image name, remove the version part for easy use
mv /boot/vmlinuz* /boot/vmlinuz

# use u-boot-tools to convert a valid ramfs
mkimage -A arm -T ramdisk -C none -n uInitrd -d /boot/initrd.img-* /boot/uInitrd
rm /boot/initrd.img-*

and then Ctrl+c to exit the chroot, flash the rootfs as a image, just do the following:

# create a empty rootfs image
truncate -s 2GiB rootfs.img

# make a ext4 file system
mkfs.ext4 rootfs.img

# create a folder to mound the image
mkdir -p imgcontent
sudo mount rootfs.img imgcontent

# copy content into it
sudo cp -rfp rootfs/* imgcontent

# umount
sudo umount rootfs.img

# fix the filesystem
e2fsck -f rootfs.img

# resize it
resize2fs -M rootfs.img

3) U-Boot

do some modifications to the file build/kconfigs/u-boot_rockpi4.conf

1. change the CONFIG_BOOTCOMMAND to the following(watch out, your dtb file is not the same with mine, your board is RockPi4C+):

   ext2load mmc 0:5 ${kernel_addr_r} /boot/vmlinuz;
   ext2load mmc 0:5 ${fdt_addr_r} /boot/rk3399-rock-pi-4b.dtb;
   ext2load mmc 0:5 ${ramdisk_addr_r} /boot/uInitrd;
   booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r};

2. change other settings(for example, I would like to use baudrate 115200):

   CONFIG_BAUDRATE=115200
   CONFIG_BOOTARGS="console=ttyS2,115200 root=PARTUUID=17d61bff-8fdc-4089-b675-9be21b9f6ac7 loglevel=6 rootwait"

   then run make -f rockpi4.mk u-boot to build u-boot, got two files: 1) idbloader.img 2) u-boot.itb

4) Create final image

run the following:

# calculate the size of the rootfs image
ROOTFS_IMAGE_SIZE_IN_KB=$[$(stat --printf="%s" rootfs.img)/1024+1]

# calculate final image size
FLASH_IMAGE_SIZE_IN_KB=$[$ROOTFS_IMAGE_SIZE_IN_KB+12288]

# init final image
truncate -s $[${FLASH_IMAGE_SIZE_IN_KB}+1024]KiB rockpi4.img

# parted
parted -s rockpi4.img \
        unit kiB  \
        mklabel gpt  \
        mkpart idbloader 32 4032 \
        mkpart primary fat32 4032 4096  \
        mkpart primary fat32 4096 8192 \
        mkpart uboot 8192 12288  \
        mkpart root ext4 12288 $FLASH_IMAGE_SIZE_IN_KB

# set part 5 uuid, we will use it as rootfs, must be the same with our u-boot command
sgdisk -u 5:17d61bff-8fdc-4089-b675-9be21b9f6ac7 rockpi4.img

# write idbloader
dd if=idbloader.img of=rockpi4.img bs=1kiB seek=32 conv=notrunc

# write uboot
dd if=u-boot.itb of=rockpi4.img bs=1kiB seek=8192 conv=notrunc

# write rootfs
dd if=rootfs.img of=rockpi4.img bs=1kiB seek=12288 conv=notrunc

5) Flash the image

use rkdeveloptool to flash the image onto the board, and boot the device, the boot flow should be: 1) RK3399 BootRom(flash, read only) 2) idbloader.img (aka u-boot spl) at 0x40 (by RK3399 BootROM code) 3) u-boot.itb at 0x4000 (by CONFIG_SYS_TEXT_BASE of u-boot), loads ramfs, kernel, dtb (by our custom boot command) 4) PART5 of our image as rootfs (by CONFIG_BOOTARGS of u-boot, match by PART UUID)

6) Repair the file system

after boot up, you might need to repair your filesystem, just: 1) type sudo mount -o remount rw / to remount the root 2) use parted's resizepart command to resize part 5 to fit your sd card size 3) use sudo resize2fs -f /dev/mmcblk1p5 to fix the file system 4) repair your /etc/fstab

[ivila]

Or you can just update the u-boot only, like what I do to my coolpi4b(a rk3588s device), just build a valid idbloader.img and u-boot.itb, and flash them to 0x40 and 0x4000 with yanyitech official image(their image left the begging 300MB for empty and the linux kernel built with OPTEE support). You can make your RockPi4C+ image by modify Radxa official image too.

[ThePerfectComputer]

Wow! A very throrough answer!

I did something similar to what you did above. However, you don't seem to build TFA or OPTEE-OS into the above suggested instructions

[ivila]

Wow! A very throrough answer!

I did something similar to what you did above. However, you don't seem to build TFA or OPTEE-OS into the above suggested instructions

When you use build to build u-boot (the command make -f rockpi4.mk u-boot), it will build OPTEE-OS and TFA first (by make file dependencies), see the make file dependencies and use them as BL31 (the tfa) and TEE (the optee_os) params while build uboot, see the UBOOT_EXPORTS

[github-actions[bot]]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment, otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.