Open retrage opened 1 year ago
Cool, perhaps @MrXinWang or @michael2012z could help. Maybe we could add GICv2 support to CH, not sure how intrusive that would be.
Awesome!
I will reach in Slack for the test environment.
The problem of GICv2 is that it does not support MSI. But in CH the virtio-PCI requires MSI. So enabling GICv2 is not very meaningful, too little devices can be supported.
Here is some updates: I'm working on support for the aarch64 CH using the test environment. (Thank you, @michael2012z!) With a custom Ubuntu bionic cloud image used in CH integration tests, it starts Linux kernel via GRUB:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1][440/1959]
[ 0.000000] Linux version 4.15.0-106-generic (buildd@bos02-arm64-071) (gcc ve
rsion 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #107-Ubuntu SMP Thu Jun 4 11:2
8:55 UTC 2020 (Ubuntu 4.15.0-106.107-generic 4.15.18)
[ 0.000000] Machine model: linux,dummy-virt
[ 0.000000] debug: skip boot console de-registration.
[ 0.000000] earlycon: pl11 at MMIO 0x0000000009000000 (options '')
[ 0.000000] bootconsole [pl11] enabled
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: EFI v2.80 by
[ 0.000000] efi:
[ 0.000000] NUMA: No NUMA configuration found
[ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x00000000fbffffff
]
[ 0.000000] NUMA: NODE_DATA [mem 0xf3fea900-0xf3fedbff]
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000040000000-0x00000000fbffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000403fffff]
[ 0.000000] node 0: [mem 0x0000000040400000-0x000000004047ffff]
[ 0.000000] node 0: [mem 0x0000000040480000-0x00000000405d7fff]
However, Linux boot process is aborted by unhandled kernel paging request:
[ 0.540102] Unable to handle kernel paging request at virtual address 4049457
0
[ 0.540102] Unable to handle kernel paging request at virtual address 4049457
0
[ 0.542886] Mem abort info:
[ 0.542886] Mem abort info:
[ 0.543972] ESR = 0x86000005
[ 0.543972] ESR = 0x86000005
[ 0.545142] Exception class = IABT (current EL), IL = 32 bits
[ 0.545142] Exception class = IABT (current EL), IL = 32 bits
[ 0.547437] SET = 0, FnV = 0
[ 0.547437] SET = 0, FnV = 0
[ 0.548624] EA = 0, S1PTW = 0
[ 0.548624] EA = 0, S1PTW = 0
[ 0.549820] [0000000040494570] user address but active_mm is swapper
[ 0.549820] [0000000040494570] user address but active_mm is swapper
[ 0.552303] Internal error: Oops: 86000005 [#1] SMP
[ 0.552303] Internal error: Oops: 86000005 [#1] SMP
It looks like the virtual address is in the EFI runtime services region, which is still not handled correctly.
Here is the working branch: https://github.com/retrage/rust-hypervisor-firmware/tree/aarch64-ch-support
Anyway, it's not so far from booting properly.
Update: It works with a custom Ubuntu bionic cloud image.
I noticed that Linux boot on CH/aarch64 has been broken since 88d7dd2301fa35b8aa806b59922c216a56d0a5b5. It seems to be a PCI related bug.
@retrage Are you only testing with CH or do you also have some QEMU support for aarch64?
This is the output I get when running with main
:
root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img
Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI
The PCI bus output looks very strange...if I go back to 411ca73cdf4a5b5c0eeae1759ab62a8752cc3fb2 it still looks peculiar and I have the same issue with booting
root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img
Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=2ff80000
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI
I test it with CH only.
Here is output:
cloud-hypervisor/target/release/cloud-hypervisor --kernel rust-hypervisor-firmware/target/aarch64-unknown-none/debug/hypervisor-fw.bin --disk path=/home/retrage/workloads/bionic-server-cloudimg-arm64.raw --cpus boot=1 --memory size=4G --console off --serial tty -v -v -v --log-file ch-debug.log
Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
PANIC: panicked at 'attempt to add with overflow', src/pci.rs:286:36
The PCI device enumeration looks really wrong:
This is what I get for RV64 on QEMU (which will be using the same MMIO ECAM code.)
Starting on RV64 0x0 0x9fe00000
Memory region 512MiB@0x80000000
Booting with FDT
Found PCI device vendor=1b36 device=8 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=16
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x1000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace64 address=0x0 size=0x4000
Bar: type=Unused address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=40000000 size=1000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace64 address=40004000 size=4000
Updated BARs: type=Unused address=0 size=0
Virtio block device configured. Capacity: 262144 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTRISCV64.EFI
root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img
Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=1
Found PCI device vendor=1af4 device=1042 in slot=2
Found PCI device vendor=1af4 device=1044 in slot=3
PCI Device: 0:1.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI
This issue tracks aarch64 support.
I am working on an experimental implementation of aarch64 support for Rust Hypervisor Firmware. The code is available at:
Current Status
The changes in this branch have been tested to work using QEMU aarch64 virt. Similar to x86_64, you can boot the Linux image by specifying the firmware with
-kernel
. The major changes are as follows:BootInfo
for FDT.I will propose these changes in multiple PRs.
To-Do
[ ] Implement remapping of EFI runtime servicesKnown Issues
This aarch64 support has the following issues.
Cloud Hypervisor is not yet supported.This aarch64 support does not yet support Cloud Hypervisor. This is because I do not have an aarch64 machine that supports GICv3 or later and cannot test it. Please let me know if there is a good test environment available.Update: It works with a custom Ubuntu bionic cloud image.
objcopy is required as post-processing after build.QEMU aarch64 virt has an emulator loader that behaves differently depending on the type of binary passed with
-kernel
. When the binary is not ELF, QEMU executes the binary with the first address of the FDT passed in the x0 register.You need to run objcopy as a post-processing step to convert it to raw binary, as follows:This post process is not needed by generating as a raw binary at link time. See eaed0717b2db4deb9e48a0e2b0ce447db56b4dcf.Run QEMU with the binary as follows: