retrage commented 1 year ago

This issue tracks aarch64 support.

I am working on an experimental implementation of aarch64 support for Rust Hypervisor Firmware. The code is available at:

QEMU aarch64 virt: https://github.com/retrage/rust-hypervisor-firmware/tree/aarch64-virt-support
Cloud Hypervisor: https://github.com/retrage/rust-hypervisor-firmware/tree/aarch64-ch-support

Current Status

The changes in this branch have been tested to work using QEMU aarch64 virt. Similar to x86_64, you can boot the Linux image by specifying the firmware with -kernel. The major changes are as follows:

Add target and linker scripts for aarch64.
Add BootInfo for FDT.
Support for PCI configuration space access using MMIO on aarch64
Add virtio-mmio driver (only for QEMU)

I will propose these changes in multiple PRs.

To-Do

~~[ ] Implement remapping of EFI runtime services~~
[x] Add initial CH support
[x] Split architecture dependent code for multiple architecture support (#203)
[x] Improve EFI compatibility for aarch64/Linux boot (#204)
[x] Add initial aarch64 support (#205)
[x] Add testing (#205)
[x] Add CI builds (#205)
[x] Support vanilla Ubuntu cloud images (#262)
[x] Add integration testing (#267)
[ ] Update documents

Known Issues

This aarch64 support has the following issues.

Cloud Hypervisor is not yet supported.

This aarch64 support does not yet support Cloud Hypervisor. This is because I do not have an aarch64 machine that supports GICv3 or later and cannot test it. Please let me know if there is a good test environment available.

Update: It works with a custom Ubuntu bionic cloud image.

objcopy is required as post-processing after build.

QEMU aarch64 virt has an emulator loader that behaves differently depending on the type of binary passed with -kernel. When the binary is not ELF, QEMU executes the binary with the first address of the FDT passed in the x0 register. ~~You need to run objcopy as a post-processing step to convert it to raw binary, as follows:~~ This post process is not needed by generating as a raw binary at link time. See eaed0717b2db4deb9e48a0e2b0ce447db56b4dcf.

Run QEMU with the binary as follows:

qemu-system-aarch64 \
  -machine virt \
  -cpu cortex-a53 \
  -m 8G \
  -nographic \
  -serial mon:stdio \
  -drive id=disk,file=$(AARCH64_IMG),if=none \
  -device virtio-blk-pci,drive=disk,disable-legacy=on \
  -kernel target/aarch64-unknown-none/debug/hypervisor-fw

rbradford commented 1 year ago

Cool, perhaps @MrXinWang or @michael2012z could help. Maybe we could add GICv2 support to CH, not sure how intrusive that would be.

michael2012z commented 1 year ago

Awesome!

I will reach in Slack for the test environment.

The problem of GICv2 is that it does not support MSI. But in CH the virtio-PCI requires MSI. So enabling GICv2 is not very meaningful, too little devices can be supported.

retrage commented 1 year ago

Here is some updates: I'm working on support for the aarch64 CH using the test environment. (Thank you, @michael2012z!) With a custom Ubuntu bionic cloud image used in CH integration tests, it starts Linux kernel via GRUB:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x413fd0c1][440/1959]
[    0.000000] Linux version 4.15.0-106-generic (buildd@bos02-arm64-071) (gcc ve
rsion 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)) #107-Ubuntu SMP Thu Jun 4 11:2
8:55 UTC 2020 (Ubuntu 4.15.0-106.107-generic 4.15.18)                           
[    0.000000] Machine model: linux,dummy-virt                                  
[    0.000000] debug: skip boot console de-registration.                        
[    0.000000] earlycon: pl11 at MMIO 0x0000000009000000 (options '')           
[    0.000000] bootconsole [pl11] enabled                                       
[    0.000000] efi: Getting EFI parameters from FDT:                            
[    0.000000] efi: EFI v2.80 by                                                
[    0.000000] efi:                                                             
[    0.000000] NUMA: No NUMA configuration found                                
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x00000000fbffffff
]                                                                               
[    0.000000] NUMA: NODE_DATA [mem 0xf3fea900-0xf3fedbff]                      
[    0.000000] Zone ranges:                                                     
[    0.000000]   DMA      [mem 0x0000000040000000-0x00000000fbffffff]           
[    0.000000]   Normal   empty                                                 
[    0.000000] Movable zone start for each node                                 
[    0.000000] Early memory node ranges                                         
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000403fffff]
[    0.000000]   node   0: [mem 0x0000000040400000-0x000000004047ffff]
[    0.000000]   node   0: [mem 0x0000000040480000-0x00000000405d7fff]

However, Linux boot process is aborted by unhandled kernel paging request:

[    0.540102] Unable to handle kernel paging request at virtual address 4049457
0                                                                               
[    0.540102] Unable to handle kernel paging request at virtual address 4049457
0                                                                               
[    0.542886] Mem abort info:                                                  
[    0.542886] Mem abort info:                                                  
[    0.543972]   ESR = 0x86000005                                               
[    0.543972]   ESR = 0x86000005                                               
[    0.545142]   Exception class = IABT (current EL), IL = 32 bits              
[    0.545142]   Exception class = IABT (current EL), IL = 32 bits              
[    0.547437]   SET = 0, FnV = 0                                               
[    0.547437]   SET = 0, FnV = 0                                               
[    0.548624]   EA = 0, S1PTW = 0                                              
[    0.548624]   EA = 0, S1PTW = 0
[    0.549820] [0000000040494570] user address but active_mm is swapper
[    0.549820] [0000000040494570] user address but active_mm is swapper
[    0.552303] Internal error: Oops: 86000005 [#1] SMP
[    0.552303] Internal error: Oops: 86000005 [#1] SMP

It looks like the virtual address is in the EFI runtime services region, which is still not handled correctly.

Here is the working branch: https://github.com/retrage/rust-hypervisor-firmware/tree/aarch64-ch-support

Anyway, it's not so far from booting properly.

Update: It works with a custom Ubuntu bionic cloud image.

retrage commented 1 year ago

I noticed that Linux boot on CH/aarch64 has been broken since 88d7dd2301fa35b8aa806b59922c216a56d0a5b5. It seems to be a PCI related bug.

rbradford commented 1 year ago

@retrage Are you only testing with CH or do you also have some QEMU support for aarch64?

rbradford commented 1 year ago

This is the output I get when running with main:

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img 

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI

The PCI bus output looks very strange...if I go back to 411ca73cdf4a5b5c0eeae1759ab62a8752cc3fb2 it still looks peculiar and I have the same issue with booting

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10
Found PCI device vendor=0 device=0 in slot=11
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=2ff80000
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Bar: type=MemorySpace32 address=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI

retrage commented 1 year ago

I test it with CH only.

retrage commented 1 year ago

Here is output:

cloud-hypervisor/target/release/cloud-hypervisor --kernel rust-hypervisor-firmware/target/aarch64-unknown-none/debug/hypervisor-fw.bin --disk path=/home/retrage/workloads/bionic-server-cloudimg-arm64.raw --cpus boot=1 --memory size=4G --console off --serial tty -v -v -v --log-file ch-debug.log

Booting with FDT                                                                
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=0 device=0 in slot=1       
Found PCI device vendor=8086 device=d57 in slot=2
Found PCI device vendor=0 device=0 in slot=3       
Found PCI device vendor=8086 device=d57 in slot=4
Found PCI device vendor=0 device=0 in slot=5       
Found PCI device vendor=8086 device=d57 in slot=6
Found PCI device vendor=0 device=0 in slot=7       
Found PCI device vendor=8086 device=d57 in slot=8
Found PCI device vendor=0 device=0 in slot=9
Found PCI device vendor=8086 device=d57 in slot=10                  
Found PCI device vendor=0 device=0 in slot=11                                  
Found PCI device vendor=8086 device=d57 in slot=12
Found PCI device vendor=0 device=0 in slot=13
Found PCI device vendor=8086 device=d57 in slot=14
Found PCI device vendor=0 device=0 in slot=15
Found PCI device vendor=1af4 device=1042 in slot=16
Found PCI device vendor=0 device=0 in slot=17
Found PCI device vendor=1af4 device=1042 in slot=18
Found PCI device vendor=0 device=0 in slot=19
Found PCI device vendor=1af4 device=1042 in slot=20
Found PCI device vendor=0 device=0 in slot=21
Found PCI device vendor=1af4 device=1042 in slot=22
Found PCI device vendor=0 device=0 in slot=23
Found PCI device vendor=1af4 device=1042 in slot=24
Found PCI device vendor=0 device=0 in slot=25
Found PCI device vendor=1af4 device=1042 in slot=26
Found PCI device vendor=0 device=0 in slot=27
Found PCI device vendor=1af4 device=1042 in slot=28
Found PCI device vendor=0 device=0 in slot=29
Found PCI device vendor=1af4 device=1042 in slot=30
Found PCI device vendor=0 device=0 in slot=31
PCI Device: 0:16.0 1af4:1042
PANIC: panicked at 'attempt to add with overflow', src/pci.rs:286:36

rbradford commented 1 year ago

The PCI device enumeration looks really wrong:

This is what I get for RV64 on QEMU (which will be using the same MMIO ECAM code.)

Starting on RV64 0x0 0x9fe00000 
Memory region 512MiB@0x80000000

Booting with FDT
Found PCI device vendor=1b36 device=8 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=16
PCI Device: 0:16.0 1af4:1042
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x1000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace64 address=0x0 size=0x4000
Bar: type=Unused address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=40000000 size=1000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace64 address=40004000 size=4000
Updated BARs: type=Unused address=0 size=0
Virtio block device configured. Capacity: 262144 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTRISCV64.EFI

rbradford commented 1 year ago

233 fixes the erroneous bus reporting:

root@ci-cloud-hypervisor-arm64-02:~/testing.rbradford/cloud-hypervisor# target/debug/cloud-hypervisor --serial tty --console off --kernel ~/testing.rbradford/rust-hypervisor-firmware/target/aarch64-unknown-none/release/hypervisor-fw --disk path=../jammy-server-cloudimg-arm64.raw --disk path=/tmp/ubuntu-cloudinit.img

Booting with FDT
Found PCI device vendor=8086 device=d57 in slot=0
Found PCI device vendor=1af4 device=1042 in slot=1
Found PCI device vendor=1af4 device=1042 in slot=2
Found PCI device vendor=1af4 device=1044 in slot=3
PCI Device: 0:1.0 1af4:1042
Bar: type=MemorySpace32 address=0x2ff80000 size=0x80000
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Bar: type=MemorySpace32 address=0x0 size=0x0
Updated BARs: type=MemorySpace32 address=2ff80000 size=80000
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Updated BARs: type=MemorySpace32 address=0 size=0
Virtio block device configured. Capacity: 4612096 sectors
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: \EFI\BOOT\BOOTAA64.EFI

cloud-hypervisor / rust-hypervisor-firmware

aarch64 Support #198

Current Status

To-Do

Known Issues

Cloud Hypervisor is not yet supported.

objcopy is required as post-processing after build.

233 fixes the erroneous bus reporting: