QEMU 4.0 virtio issue with OP-TEE 3.5+

I'm unable to run a fully ARM64 virtualized rootfs image with QEMU after updating OP-TEE OS to 3.5, which seems to be causing virtio (block) related issues.

QEMU starts fine, booting Linux (5.0-based) up to the point where it tries to mount the rootfs and start the main system, at which point it just freezes completely. This same setup and same image works fine when using OP-TEE 3.4, but not with 3.5+.

The error message I see on the QEMU side when booting the system:

$ qemu-system-aarch64 -device virtio-net-device,netdev=net0,mac=52:54:00:12:35:02 \
-netdev user,id=net0,hostfwd=tcp::2222-:22 \
-drive if=virtio,file=lmp-gateway-image-qemuarm64.wic,format=raw \
-no-reboot -no-acpi -bios bl1.bin -d unimp -semihosting-config enable,target=native \
-nographic -machine virt,secure=on -cpu cortex-a57 -m 1057 -smp 2 -S \
-serial telnet:127.0.0.1:4444,server,nowait -serial telnet:127.0.0.1:4445,server,nowait
QEMU 4.0.0 monitor - type 'help' for more information
(qemu) c
(qemu) qemu-system-aarch64: virtio: bogus descriptor or out of resources

Bisecting OP-TEE OS after 3.4 got me to commit e9495e2a350d2cf8aee2da6f1e22651143cd4270, and I can confirm that even current HEAD works fine after reverting this change.

Looks like the dt reserve for tzdram is somehow confusing the kernel/qemu, only thing I'm not yet sure is why this is happening, since the tzdram offset seems correct:

out/arm/include/generated/conf.h:#define CFG_TZDRAM_SIZE 0x00f00000
out/arm/include/generated/conf.h:#define CFG_TZDRAM_START 0x0e100000

And from QEMU: https://github.com/qemu/qemu/blob/master/hw/arm/virt.c#L140

The only thing I noticed is that there is no need to reserve that region with QEMU as it is already set and initialized via https://github.com/qemu/qemu/blob/master/hw/arm/virt.c#L1300

Not sure if a conflict from qemu/linux side yet, but wanted to report the issue in case anyone is also able to reproduce it.

Hi @ricardosalveti,

I have not looked in details but at least I have tried to use QEMU v4.0.0 with our latest code, and I had no problem running xtest. There is just a weird message about pflash_write that we don't have with older QEMU. I'm using the QEMU v8 environment described in https://optee.readthedocs.io/building/devices/qemu.html#qemu-v8. Perhaps it's your kernel? Have you tried https://github.com/linaro-swg/linux branch optee? Anyway here are more details on my setup:

$ make run
[...]
QEMU 4.0.0 monitor - type 'help' for more information
(qemu) c
(qemu) pflash_write: Write to buffer emulation is flawed
pflash_write: Write to buffer emulation is flawed

=================

$ xtest
[...]
24078 subtests of which 0 failed
95 test cases of which 0 failed
0 test cases were skipped
TEE test application done!

=================

$ repo manifest -r
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote fetch="https://github.com" name="github"/>

  <default remote="github" revision="master"/>

  <project name="ARM-software/arm-trusted-firmware.git" path="arm-trusted-firmware" revision="34efb683e32254b8c325ac3071c5776d243a7b99"/>
  <project name="OP-TEE/build.git" path="build" revision="1a16301b76243964f60630724fb0e75c2ae64aae" upstream="master">
    <linkfile dest="build/Makefile" src="qemu_v8.mk"/>
    <linkfile dest="build/gdb" src="../toolchains/aarch64/bin/aarch64-linux-gnu-gdb"/>
  </project>
  <project name="OP-TEE/optee_client.git" path="optee_client" revision="1cdf49d9259ad83c3fbdf416e5ea223a18a28da8" upstream="master"/>
  <project name="OP-TEE/optee_os.git" path="optee_os" revision="f23f99182476abb375aa9e285e6e2a854a4ac938" upstream="master"/>
  <project name="OP-TEE/optee_test.git" path="optee_test" revision="83a30caee8e9e9c7aff75ef12709b625506349ff" upstream="master"/>
  <project name="buildroot/buildroot.git" path="buildroot" revision="95942f5fcd35d783a49adce621ccf33480f1c88c"/>
  <project name="linaro-swg/linux.git" path="linux" revision="9823b258b332b4ac98e05fa23448bbc9e937b24c"/>
  <project name="linaro-swg/optee_benchmark.git" path="optee_benchmark" revision="dfad7d2336b0aaf938b88b32f1df00d1cd9ccfa0" upstream="master"/>
  <project name="linaro-swg/optee_examples.git" path="optee_examples" revision="292da2fd8db0176e0e96989268c63ef9ab910a6c" upstream="master"/>
  <project name="linaro-swg/soc_term.git" path="soc_term" revision="5493a6e7c264536f5ca63fe7511e5eed991e4f20"/>
  <project clone-depth="1" name="qemu/qemu.git" path="qemu" revision="131b9a05705636086699df15d4a6d328bb2585e8" upstream="refs/tags/v3.1.0-rc3"/>
  <project name="tianocore/edk2.git" path="edk2" revision="dd4cae4d82c7477273f3da455084844db5cca0c0"/>
</manifest>

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.

OP-TEE / optee_os

QEMU 4.0 virtio issue with OP-TEE 3.5+ #3104