Guest has not initialized the display on pve8 && No IOMMU detected

[wanlce]

NanoPC-T6 rk3588
OS: Armbian (23.08.0-trunk) aarch64 Bookworm

related https://github.com/jiangcuo/Proxmox-Arm64/issues/20 https://github.com/jiangcuo/Proxmox-Arm64/issues/28 I've read the wiki in detail and the relevant issues , made the appropriate changes, but they don't work for me, my VM startup keeps getting stuck at Guest has not initialized the display (yet), and cpu 100%.

nanopct6:pve-edk2-firmware:# qm terminal 100
unable to find a serial interface
nanopct6:pve-edk2-firmware:# dpkg -l | grep pve-edk2
ii  pve-edk2-firmware                    3.20220526-1                            all          edk2 based UEFI firmware modules for virtual machines
nanopct6:pve-edk2-firmware:# ll  /usr/share/pve-edk2-firmware/AAVMF_CODE.fd
-rw-r--r-- 1 root root 64M Jul 19  2022 /usr/share/pve-edk2-firmware/AAVMF_CODE.fd
nanopct6:pve-edk2-firmware:# ll  /usr/share/pve-edk2-firmware/AAVMF_CODE.ms.fd
lrwxrwxrwx 1 root root 13 Aug 28 02:47 /usr/share/pve-edk2-firmware/AAVMF_CODE.ms.fd -> AAVMF_CODE.fd

qm showcmd

nanopct6:pve-edk2-firmware:# qm showcmd 100
/usr/bin/kvm -id 100 -name 'openwrt,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=6ac35558-c03c-4c51-8c2e-cb8151fd96d5' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' -cpu host -m 4096 -object 'iothread,id=iothread-virtioscsi0' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f' -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pcie.0,addr=0x5' -device 'qemu-xhci,id=qemu-xhci' -device 'usb-tablet,id=tablet,bus=qemu-xhci.0,port=1' -device 'usb-kbd,id=keyboard,bus=qemu-xhci.0,port=2' -readconfig /usr/share/qemu-server/pve-aarch64.cfg -device 'virtio-gpu,id=vga,bus=pcie.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' -device 'virtio-serial,id=qga0,bus=pcie.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-serial,id=spice,bus=pcie.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -spice 'tls-port=61001,addr=::1,tls-ciphers=HIGH,seamless-migration=on' -device 'virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c0c49f78d50' -device 'virtio-scsi-pci,id=virtioscsi0,bus=pcie.3,addr=0x1,iothread=iothread-virtioscsi0' -drive 'file=/var/lib/vz/images/100/vm-100-disk-0.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=A6:9D:CF:43:9B:4B,netdev=net0,bus=pcie.0,addr=0xb,id=net0,rx_queue_size=1024,tx_queue_size=1024,bootindex=101' -netdev 'type=tap,id=net1,ifname=tap100i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=9A:A4:37:7A:89:86,netdev=net1,bus=pcie.1,addr=0x2,id=net1,rx_queue_size=1024,tx_queue_size=1024,bootindex=102' -machine 'type=virt+pve0,gic-version=host' -machine 'virt,gic-version=host'

pveversion --verbos

nanopct6:pve-edk2-firmware:# pveversion --verbos
proxmox-ve: 8.0.1 (running kernel: 5.10.160-legacy-rk35xx)
pve-manager: 8.0.3-1 (running version: 8.0.3-1/a8e18f85609c0f55)
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx3
libjs-extjs: 7.0.0-3
libknet1: 1.25-1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.4
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20220526-1
pve-firewall: 5.0.3
pve-firmware: not correctly installed
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6-1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0

Thanks in advance!

[albertogrimana]

Hi!!

It's strange.. Your version of pve-edk package is good This is my configuration in one virtual machine

Other configuration that you need review is 'CPU affinity' inside processors option

it's very important that set it for little (0,1,2,3) or big processor (4,5,6,7) because it's not possible to mix them causing the problem of 100% of one or more cpu

[jiangcuo]

@wanlce Hi, Arm64 VM need UEFI bios. Change bios to UEFI and try again.

[jiangcuo]

@albertogrimana Hello,I notice that you enabled the security boot in your vm config. Can vm boot to os with secboot on？

[albertogrimana]

Depends of your SO I use Alpine Linux and Debian and work with any problem but you need activate If not activate vm don't boot as @jiangcuo said

Regards

[wanlce]

@jiangcuo @albertogrimana Thanks a lot for the guidance. This time, I added CPU affinity and set the BIOS to UEFI. After making these changes, the system can now start normally.

Additionally, it seems that there are some issues with my IOMMU. Do I need to make any specific configurations?

nanopct6:~:# dmesg | grep -i iommu
[    2.491866] Kernel command line: root=UUID=c9e8e738-d77e-4503-80ea-faf2ecea886a rootwait rootfstype=ext4 splash=verbose console=ttyFIQ0 console=tty1 consoleblank=0 loglevel=1 ubootpart= usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u   iommu=on cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1 earlycon=uart8250,mmio32,0xfeb50000 coherent_pool=1m irqchip.gicv3_pseudo_nmi=0
[    3.918449] iommu: Default domain type: Translated 
[    6.086919] mpp_vdpu2 fdb50400.vdpu: Adding to iommu group 1
[    6.087727] mpp_vepu2 fdba0000.jpege-core: Adding to iommu group 5
[    6.088213] mpp_vepu2 fdba4000.jpege-core: Adding to iommu group 6
[    6.088675] mpp_vepu2 fdba8000.jpege-core: Adding to iommu group 7
[    6.089152] mpp_vepu2 fdbac000.jpege-core: Adding to iommu group 8
[    6.089814] mpp-iep2 fdbb0000.iep: Adding to iommu group 9
[    6.090434] mpp_jpgdec fdb90000.jpegd: Adding to iommu group 4
[    6.091395] mpp_rkvdec2 fdc38100.rkvdec-core: Adding to iommu group 12
[    6.092166] mpp_rkvdec2 fdc48100.rkvdec-core: Adding to iommu group 13
[    6.093185] mpp_rkvenc2 fdbd0000.rkvenc-core: Adding to iommu group 10
[    6.093764] mpp_rkvenc2 fdbe0000.rkvenc-core: Adding to iommu group 11
[    6.094906] mpp_av1dec av1d-master: av1_iommu_of_xlate,784
[    6.094923] av1_iommu_probe_device,736, consumer : av1d-master, supplier : fdca0000.iommu
[    6.094954] mpp_av1dec av1d-master: Adding to iommu group 15
[    6.112801] rockchip-vop2 fdd90000.vop: Adding to iommu group 14
[    6.983692] rga3_core0 fdb60000.rga: Adding to iommu group 2
[    6.984908] rga3_core1 fdb70000.rga: Adding to iommu group 3
[    6.986625] rga_iommu: IOMMU binding successfully, default mapping core[0x1]
[    7.042886] RKNPU fdab0000.npu: Adding to iommu group 0
[    7.043013] RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[    7.146164]     iommu=on

nanopct6:~:# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

[albertogrimana]

Hi

Sorry but I use a Orange Pi 5 and have RK3588S SOC and I cannot try IOMMU Can you execute this comand?

pvesh get /nodes/"namenode"/hardware/pci --pci-class-blacklist ""

Where namenode I think is nanopct6 in your case, is the name of node in proxmox web ui

Regards

[Dexogen]

Hello,

I have same problem with Orange PI 5 Plus.

lsb_release -a

```bash Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm Linux dex-opi-2 5.10.110-rockchip-rk3588 #1.0.6 SMP Thu Jun 1 11:43:16 CST 2023 aarch64 GNU/Linux ```

pveversion --verbos

```yaml proxmox-ve: 8.0.1 (running kernel: 5.10.110-rockchip-rk3588) pve-manager: 8.0.3-1 (running version: 8.0.3-1/a8e18f85609c0f55) ceph: 17.2.6-pve1+3 ceph-fuse: 17.2.6-pve1+3 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown: residual config ifupdown2: 3.2.0-1+pmx2 libjs-extjs: 7.0.0-3 libknet1: 1.25-1 libproxmox-acme-perl: 1.4.6 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.0 libpve-access-control: 8.0.4 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.0.6 libpve-guest-common-perl: 5.0.3 libpve-http-server-perl: 5.0.4 libpve-rs-perl: 0.8.3 libpve-storage-perl: 8.0.2 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-2 proxmox-backup-client: 3.0.1-1 proxmox-backup-file-restore: 3.0.1-1 proxmox-kernel-helper: 8.0.2 proxmox-mail-forward: 0.2.0 proxmox-mini-journalreader: 1.4.0 proxmox-widget-toolkit: 4.0.6 pve-cluster: 8.0.2 pve-container: 5.0.4 pve-docs: 8.0.4 pve-edk2-firmware: 3.20230228-4 pve-firewall: 5.0.3 pve-firmware: not correctly installed pve-ha-manager: 4.0.2 pve-i18n: 3.0.5 pve-qemu-kvm: 8.0.2-3 pve-xtermjs: 4.16.0-3 qemu-server: 8.0.6-1 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0 zfsutils-linux: 2.1.12-pve1 ```

cat /etc/pve/qemu-server/125.conf

```yaml affinity: 0,1,2,3 agent: 1,fstrim_cloned_disks=1 bios: ovmf boot: order=virtio0;scsi0 cipassword: $5$j9EkIqfs$lKKPGXs2LBoQ0suaep9FI.JYIhWFgYUqwt6rC8Fz339 ciuser: ubuntu cores: 4 cpu: host efidisk0: ceph:vm-125-disk-0,efitype=4m,pre-enrolled-keys=1,size=64M hotplug: disk,network,usb ipconfig0: ip=dhcp memory: 8192 meta: creation-qemu=8.0.2,ctime=1693562164 name: test-vm net0: virtio=2A:9B:15:FE:0E:E1,bridge=vmbr0 numa: 0 scsi0: ceph:vm-125-cloudinit,media=cdrom,size=4M scsihw: virtio-scsi-single serial0: socket smbios1: uuid=49d9c5b7-8e7a-479e-998e-99ab4a6c45d9 sockets: 1 vga: serial0 virtio0: ceph:vm-125-disk-1,discard=on,iothread=1,size=4G ```

It seems that I set the settings as indicated above, but I still get the message Guest has not initialized the display (yet). in the case of "Default" graphics card. Or an empty output in the case of serial0 port: starting serial terminal on interace serial0.

I have tried booting from the Ubuntu-arm64 iso image and via importing the cloud-init image. The result is the same. What am I missing?

Thanks

UPD. I'm sorry, I found the solution in the related issues.

apt download  pve-edk2-firmware=3.20220526-1 
dpkg -i pve-edk2-firmware_3.20220526-1_all.deb

echo 'Package: pve-edk2-firmware
Pin: version 3.20220526-1
Pin-Priority: 999' > /etc/apt/preferences.d/pve-edk2-firmware

[jiangcuo]

Your device does not have a PCIe device. I have no idea about platform devices passthrough now.

[SuperKali]

I have the same problem: proxmox-ve: 8.0.1 (running kernel: 6.5.0-rc1-collabora-rockchip-rk3588) pve-manager: 8.0.3-1 (running version: 8.0.3-1/a8e18f85609c0f55) ceph-fuse: 17.2.6-pve1+3 corosync: 3.1.7-pve3 criu: 3.17.1-2 glusterfs-client: 10.3-5 ifupdown: residual config ifupdown2: 3.2.0-1+pmx2 libjs-extjs: 7.0.0-3 libknet1: 1.25-1 libproxmox-acme-perl: 1.4.6 libproxmox-backup-qemu0: 1.4.0 libproxmox-rs-perl: 0.3.0 libpve-access-control: 8.0.4 libpve-apiclient-perl: 3.3.1 libpve-common-perl: 8.0.6 libpve-guest-common-perl: 5.0.3 libpve-http-server-perl: 5.0.4 libpve-rs-perl: 0.8.3 libpve-storage-perl: 8.0.2 libspice-server1: 0.15.1-1 lvm2: 2.03.16-2 lxc-pve: 5.0.2-4 lxcfs: 5.0.3-pve3 novnc-pve: 1.4.0-2 proxmox-backup-client: 3.0.1-1 proxmox-backup-file-restore: 3.0.1-1 proxmox-kernel-helper: 8.0.2 proxmox-mail-forward: 0.2.0 proxmox-mini-journalreader: 1.4.0 proxmox-widget-toolkit: 4.0.6 pve-cluster: 8.0.2 pve-container: 5.0.4 pve-docs: 8.0.4 pve-edk2-firmware: 3.20230228-4 pve-firewall: 5.0.3 pve-firmware: not correctly installed pve-ha-manager: 4.0.2 pve-i18n: 3.0.5 pve-qemu-kvm: 8.0.2-3 pve-xtermjs: 4.16.0-3 qemu-server: 8.0.6-1 smartmontools: 7.3-pve1 spiceterm: 3.3.0 swtpm: 0.8.0+pve1 vncterm: 1.8.0

@wanlce why you're using a 5.10 kernel, i see online armbian for nanopc t6 have only the kernel 6.5, i have a problem to find a kernel with kvm enabled, dietpi have no kvm enabled.

[wanlce]

@SuperKali use armbian/build to build and select legacy

[SuperKali]

@wanlce with legacy kernel i have much problem than 6.5 lol, it better 6.5 but iommu seems not work

root@t6:/dev# dmesg | grep -i iommu [ 0.045098] iommu: Default domain type: Translated [ 0.045102] iommu: DMA domain TLB invalidation policy: strict mode

and the driver card seems disabled..

[SuperKali]

@wanlce have you tried the kernel 6.5, this with proxmox arm64 works fine, only not work with pcie passthrough, or you've fixed it?

[wanlce]

@wanlce have you tried the kernel 6.5, this with proxmox arm64 works fine, only not work with pcie passthrough, or you've fixed it?

I tried 6.5, but the experience was not very good. The pci passthrough still doesn't work, and more importantly I need to boot from nvme, which doesn't have a corresponding sfc node in the 6.5 device tree. https://github.com/armbian/linux-rockchip/commit/7c4108a922a0c89ecb0acf2bb0603a998b7b4e43

[SuperKali]

@wanlce have you tried the kernel 6.5, this with proxmox arm64 works fine, only not work with pcie passthrough, or you've fixed it?

I tried 6.5, but the experience was not very good. The pci passthrough still doesn't work, and more importantly I need to boot from nvme, which doesn't have a corresponding sfc node in the 6.5 device tree. armbian/linux-rockchip@7c4108a

The nvme works fine only emmc have problems.

[cantalupo555]

@wanlce have you tried the kernel 6.5, this with proxmox arm64 works fine, only not work with pcie passthrough, or you've fixed it?

I tried 6.5, but the experience was not very good. The pci passthrough still doesn't work, and more importantly I need to boot from nvme, which doesn't have a corresponding sfc node in the 6.5 device tree. armbian/linux-rockchip@7c4108a

TFcard doesn't work well. Basic activities such as updating packages become very time-consuming. You must at least have NVMe.

[bbadm1108]

Same issue took place after I installed the PVE 8.0.3 on Ubuntu/Debian/Armbian (bookworm version, tried each one), the whole installation looked smooth with pve console normally displayed , but any VMs setup couldn't boot with output (Guest has not initialized the display or Display out is not activated), whatever how to set up the display. It was strictly complied with the VM guidance and seemed stuck somewhere with CPU full utilization and no IP address found. Tried Dexogen's solution above without progress. BTW, my device is OrangePi 5 Plus.

[wanlce]

@bbadm1108 Try this.

wget http://download.proxmox.com/debian/pve/dists/bullseye/pve-no-subscription/binary-amd64/pve-edk2-firmware_3.20220526-1_all.deb
dpkg -i pve-edk2-firmware_3.20220526-1_all.deb

# create symboly link
cd /usr/share/pve-edk2-firmware
ln -s AAVMF_CODE.fd AAVMF_CODE.ms.fd

[bbadm1108]

pvesh get /nodes/"namenode"/hardware/pci --pci-class-blacklist ""

Thank you wanlce!

I tried your solution and finally got it working with console displayed...an encouraging moving!

next step is to figure out the pci device passthrough, which is another obstacle upfront with NO IOMMU detected issue.