search

seL4 / camkes-vm

Virtual Machine built as a CAmkES component.
Other
23 stars 35 forks source link

qemu-arm-virt turns on smp and reports an error when running #26

Closed polijh closed 2 years ago

polijh commented 2 years ago

I followed the tutorial at https://github.com/seL4/camkes-vm-examples and tried to use qemu to simulate the environment of arm64 to run sel4 and start the linux virtual machine. At present, the single-core case can run correctly, but the multi-core error is reported: This is my operation

1.
bst@bst-Vostro-3671:~/chenyu/sel4-arm64/build$ ../init-build.sh -DCAMKES_VM_APP=vm_minimal -DPLATFORM=qemu-arm-virt -DNUM_NODES=2
loading initial cache file /home/bst/chenyu/sel4-arm64/projects/vm-examples/settings.cmake
-- Set platform details from PLATFORM=qemu-arm-virt
--   KernelPlatform: qemu-arm-virt
-- Found seL4: /home/bst/chenyu/sel4-arm64/kernel  
-- Default cpu specified for virt board: cortex-a53
-- QEMU MEMORY is: 2048
-- 
-- Default cpu specified for virt board: cortex-a53
-- QEMU MEMORY is: 2048
-- 
-- Found GCC with prefix aarch64-linux-gnu-
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/aarch64-linux-gnu-gcc
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found camkes-arm-vm: /home/bst/chenyu/sel4-arm64/projects/vm  
-- Found camkes-tool: /home/bst/chenyu/sel4-arm64/projects/camkes-tool  
-- Found global-components: /home/bst/chenyu/sel4-arm64/projects/global-components  
-- Found camkes-vm: /home/bst/chenyu/sel4-arm64/projects/vm  
-- Found camkes-vm-images: /home/bst/chenyu/sel4-arm64/projects/camkes-vm-images  
-- Found sel4_projects_libs: /home/bst/chenyu/sel4-arm64/projects/seL4_projects_libs  
-- Found elfloader-tool: /home/bst/chenyu/sel4-arm64/tools/seL4/elfloader-tool  
-- Found musllibc: /home/bst/chenyu/sel4-arm64/projects/musllibc  
-- Found util_libs: /home/bst/chenyu/sel4-arm64/projects/util_libs  
-- Found seL4_libs: /home/bst/chenyu/sel4-arm64/projects/seL4_libs  
-- Found projects_libs: /home/bst/chenyu/sel4-arm64/projects/projects_libs  
-- Found capdl: /home/bst/chenyu/sel4-arm64/projects/capdl  
-- QEMU MEMORY is: 2048
-- 
-- /home/bst/chenyu/sel4-arm64/build/kernel/gen_headers/plat/machine/devices_gen.h is out of date. Regenerating from DTB...
-- CPIO test cpio_reproducible_flag PASSED
-- Detecting cached version of: capDL-tool
-- Found Git: /usr/bin/git (found version "2.25.1") 
--   Found valid cache entry for capDL-tool
-- Found sel4runtime: /home/bst/chenyu/sel4-arm64/projects/sel4runtime  
-- Performing Test compiler_arch_test
-- Performing Test compiler_arch_test - Success
-- libmuslc architecture: 'aarch64' (from KernelSel4Arch 'aarch64')
-- Detecting cached version of: musllibc
--   Found valid cache entry for musllibc
BenchUtiliz not available, as KernelArmExportPMUUser is OFF
-- Found camkes-vm-linux: /home/bst/chenyu/sel4-arm64/projects/vm-linux  
-- /home/bst/chenyu/sel4-arm64/build/ast.pickle is out of date. Regenerating...
-- /home/bst/chenyu/sel4-arm64/build/camkes-gen.cmake is out of date. Regenerating...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/bst/chenyu/sel4-arm64/build
2
bst@bst-Vostro-3671:~/chenyu/sel4-arm64/build$ ninja 
[85/376] Building C object sel4_projects_libs/libsel4dma/CMakeFiles/sel4dma.dir/src/dma.c.obj
/home/bst/chenyu/sel4-arm64/projects/seL4_projects_libs/libsel4dma/src/dma.c:32:2: warning: #warning Unknown platform. DMA alignment defaulting to 32 bytes. [-Wcpp]
   32 | #warning Unknown platform. DMA alignment defaulting to 32 bytes.
      |  ^~~~~~~
[326/376] Building C object CMakeFiles/capdl-loader.dir/home/bst/chenyu/sel4-arm64/projects/capdl/capdl-loader-app/src/main.c.obj
/home/bst/chenyu/sel4-arm64/projects/capdl/capdl-loader-app/src/main.c: In function ‘get_capData’:
/home/bst/chenyu/sel4-arm64/projects/capdl/capdl-loader-app/src/main.c:154:18: note: parameter passing for argument of type ‘CDL_CapData’ {aka ‘struct <anonymous>’} changed in GCC 9.1
  154 | static seL4_Word get_capData(CDL_CapData d)
      |                  ^~~~~~~~~~~
[376/376] Generating images/capdl-loader-image-arm-qemu-arm-virt
3.
bst@bst-Vostro-3671:~/chenyu/sel4-arm64/build$ ./simulate --extra-qemu-args="-smp 2"
./simulate: qemu-system-aarch64 -machine virt,virtualization=on,highmem=off,secure=off -cpu cortex-a53 -nographic  -m size=2048 -smp 2 -kernel images/capdl-loader-image-arm-qemu-arm-virt 
ELF-loader started on CPU: ARM Ltd. Cortex-A53 r0p4
  paddr=[61b48000..62fea0d7]
No DTB passed in from boot loader.
Looking for DTB in CPIO archive...found at 61cce230.
Loaded DTB from 61cce230.
   paddr=[6024c000..6024dfff]
ELF-loading image 'kernel' to 60000000
  paddr=[60000000..6024bfff]
  vaddr=[8060000000..806024bfff]
  virt_entry=8060000000
ELF-loading image 'capdl-loader' to 6024e000
  paddr=[6024e000..61756fff]
  vaddr=[400000..1908fff]
  virt_entry=408f58
Boot cpu id = 0x0, index=0
Core 1 is up with logic id 1
Enabling hypervisor MMU and paging
Jumping to kernel-image entry point...

Bootstrapping kernel
available phys memory regions: 1
  [60000000..c0000000]
reserved virt address space regions: 3
  [8060000000..806024c000]
  [806024c000..806024de1f]
  [806024e000..8061757000]
Booting all finished, dropped to user space
<<seL4(CPU 0) [decodeUntypedInvocation/205 T0x80bc013400 "rootserver" @4006f8]: Untyped Retype: Insufficient memory (1 * 2097152 bytes needed, 0 bytes available).>>
Loading Linux: 'linux' dtb: 'linux-dtb'
install_linux_devices@main.c:651 module name: map_frame_hack
install_linux_devices@main.c:651 module name: init_ram
libsel4muslcsys: Error attempting syscall 215
libsel4muslcsys: Error attempting syscall 215
load_linux@main.c:831 Failed to generate a fdt
Failed to load VM image
Halting...

I noticed that the fdt is not being generated correctly. Debugging with gdb, I found that sel4 will generate psci nodes for fdt, so I make changes:

bst@bst-Vostro-3671:~/chenyu/sel4-arm64/projects/vm/components/VM_Arm/plat_include/qemu-arm-virt/plat$ git diff vmlinux.h
diff --git a/components/VM_Arm/plat_include/qemu-arm-virt/plat/vmlinux.h b/components/VM_Arm/plat_include/qemu-arm-virt/plat/vmlinux.h
index e8ec49e..a51d06f 100644
--- a/components/VM_Arm/plat_include/qemu-arm-virt/plat/vmlinux.h
+++ b/components/VM_Arm/plat_include/qemu-arm-virt/plat/vmlinux.h
@@ -26,7 +26,7 @@ static const char *plat_keep_devices[] = {
     "/platform@c000000",
     "/pmu",
     "/flash@0",
-    "/psci",
+    //"/psci",
 };
 static const char *plat_keep_device_and_disable[] = {};
 static const char *plat_keep_device_and_subtree[] = {

Finally, I thought I entered the linux kernel, but there is a loop of printing:

ELF-loader started on CPU: ARM Ltd. Cortex-A53 r0p4
  paddr=[61b48000..62fea0d7]
No DTB passed in from boot loader.
Looking for DTB in CPIO archive...found at 61cce230.
Loaded DTB from 61cce230.
   paddr=[6024c000..6024dfff]
ELF-loading image 'kernel' to 60000000
  paddr=[60000000..6024bfff]
  vaddr=[8060000000..806024bfff]
  virt_entry=8060000000
ELF-loading image 'capdl-loader' to 6024e000
  paddr=[6024e000..61756fff]
  vaddr=[400000..1908fff]
  virt_entry=408f58
Boot cpu id = 0x0, index=0
Core 1 is up with logic id 1
Enabling hypervisor MMU and paging
Jumping to kernel-image entry point...

Bootstrapping kernel
available phys memory regions: 1
  [60000000..c0000000]
reserved virt address space regions: 3
  [8060000000..806024c000]
  [806024c000..806024de1f]
  [806024e000..8061757000]
Booting all finished, dropped to user space
^[[0m^[[30;1m<<^[[0m^[[32mseL4(CPU 0)^[[0m^[[30;1m [decodeUntypedInvocation/205 T0x80bc013400 "rootserver" @4006f8]: Untyped Retype: Insufficient memory (1 * 2097152 bytes needed, 0 bytes available).>>^[[0m
Loading Linux: 'linux' dtb: 'linux-dtb'
install_linux_devices@main.c:651 module name: map_frame_hack
install_linux_devices@main.c:651 module name: init_ram
libsel4muslcsys: Error attempting syscall 215
libsel4muslcsys: Error attempting syscall 215
_utspace_split_alloc@split.c:266 Failed to find any untyped capable of creating an object at address 0x8020000
alloc_vm_device_cap@main.c:938 Grabbing the entire cap for device memory
alloc_vm_device_cap@main.c:941 Failed to grab the entire cap
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.189+ (alisonf@shinyu-un) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #16 SMP Tue Feb 25 14:14:50 AEDT 2020
[    0.000000] Boot CPU: AArch64 Processor [410fd034]
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.0 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.0
[    0.000000] percpu: Embedded 22 pages/cpu s51608 r8192 d30312 u90112^M
[    0.000000] Detected VIPT I-cache on CPU0^M
[    0.000000] CPU features: enabling workaround for ARM erratum 845719^M
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129024^M
[    0.000000] Kernel command line:  maxcpus=2^M
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)^M
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)^M
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)^M
[    0.000000] Memory: 490360K/524288K available (7484K kernel code, 1220K rwdata, 2524K rodata, 3776K init, 543K bss, 33928K reserved, 0K cma-reserved)^M
[    0.000000] Virtual kernel memory layout:^M
[    0.000000]     modules : 0xffff000000000000 - 0xffff000008000000   (   128 MB)^M
[    0.000000]     vmalloc : 0xffff000008000000 - 0xffff7dffbfff0000   (129022 GB)^M
[    0.000000]       .text : 0xffff000008080000 - 0xffff0000087d0000   (  7488 KB)^M
[    0.000000]     .rodata : 0xffff0000087d0000 - 0xffff000008a50000   (  2560 KB)^M
[    0.000000]       .init : 0xffff000008a50000 - 0xffff000008e00000   (  3776 KB)^M
[    0.000000]       .data : 0xffff000008e00000 - 0xffff000008f31200   (  1221 KB)^M
[    0.000000]        .bss : 0xffff000008f31200 - 0xffff000008fb9034   (   544 KB)^M
[    0.000000]     fixed   : 0xffff7dfffe7fb000 - 0xffff7dfffec00000   (  4116 KB)^M
[    0.000000]     PCI I/O : 0xffff7dfffee00000 - 0xffff7dffffe00000   (    16 MB)^M
[    0.000000]     vmemmap : 0xffff7e0000000000 - 0xffff800000000000   (  2048 GB maximum)^M
[    0.000000]               0xffff7e0000000000 - 0xffff7e0000800000   (     8 MB actual)^M
[    0.000000]     memory  : 0xffff800000000000 - 0xffff800020000000   (   512 MB)^M
[    0.000000] Hierarchical RCU implementation.^M
[    0.000000]  Build-time adjustment of leaf fanout to 64.^M
[    0.000000]  RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.^M
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2^M
[    0.000000] NR_IRQS:64 nr_irqs:64 0^M
[    0.000000] >>>>>> gic cpu init | CPU 0 | cpu_mask 0x1^M
[    0.000000] GICv2m: Invalid MSI base SPI (base:0)^M
[    0.000000] arm_arch_timer: Architected cp15 timer(s) running at 62.50MHz (virt).^M
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns^M
[    0.000157] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns^M
[    0.011289] Console: colour dummy device 80x25^M
[    0.013115] console [tty0] enabled^M
[    0.013904] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=250000)^M
[    0.014740] pid_max: default: 32768 minimum: 301^M
[    0.016122] Security Framework initialized^M
[    0.016209] Yama: becoming mindful.^M
[    0.017196] AppArmor: AppArmor disabled by boot time parameter^M
[    0.017738] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes)^M
[    0.017807] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes)^M
[    0.033214] ftrace: allocating 26505 entries in 104 pages^M
[    0.208708] ASID allocator initialised with 32768 entries^M
[    0.223000] EFI services will not be available.^M
[    0.231891] Detected VIPT I-cache on CPU1^M
[    0.233481] >>>>>> gic cpu init | CPU 1 | cpu_mask 0x2^M
[    0.238295] CPU1: Booted secondary processor [410fd034]^M
[    0.239530] >>>> MAP for cpu: 0 | map: 1^M
[    0.240338] INJECTING SOFTIRQ -  multi map: 0x10000^M
[    0.244161] >>>> MAP for cpu: 0 | map: 1^M
[    0.244212] INJECTING SOFTIRQ -  multi map: 0x10000^M
[    0.245121] >>>> MAP for cpu: 1 | map: 2^M
[    0.245205] INJECTING SOFTIRQ -  multi map: 0x20000^M
[    0.256141] >>>> MAP for cpu: 0 | map: 1^M
[    0.256194] INJECTING SOFTIRQ -  multi map: 0x10000^M
[    0.256822] Brought up 2 CPUs^M
[    0.256895] SMP: Total of 2 processors activated.^M
[    0.257054] CPU features: detected feature: 32-bit EL0 Support^M
[    0.257136] CPU features: detected feature: Kernel page table isolation (KPTI)^M
[    0.257554] >>>> MAP for cpu: 1 | map: 2^M
[    0.257581] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.258930] >>>> MAP for cpu: 1 | map: 2^M
[    0.258976] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.259335] >>>> MAP for cpu: 1 | map: 2^M
[    0.259380] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.259761] >>>> MAP for cpu: 1 | map: 2^M
[    0.259805] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.260190] >>>> MAP for cpu: 1 | map: 2^M
[    0.260234] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.260615] >>>> MAP for cpu: 1 | map: 2^M
[    0.260659] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.261003] >>>> MAP for cpu: 1 | map: 2^M
[    0.261048] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.261437] >>>> MAP for cpu: 1 | map: 2^M
[    0.261482] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.261816] >>>> MAP for cpu: 1 | map: 2^M
[    0.261859] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.262184] >>>> MAP for cpu: 1 | map: 2^M
[    0.262384] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.263100] >>>> MAP for cpu: 1 | map: 2^M
[    0.263146] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.263442] >>>> MAP for cpu: 1 | map: 2^M
[    0.263486] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.263774] >>>> MAP for cpu: 1 | map: 2^M
[    0.263817] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.264123] >>>> MAP for cpu: 1 | map: 2^M
[    0.264167] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.264475] >>>> MAP for cpu: 1 | map: 2^M
[    0.264534] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.264829] >>>> MAP for cpu: 1 | map: 2^M
[    0.264873] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.265191] >>>> MAP for cpu: 1 | map: 2^M
[    0.265235] INJECTING SOFTIRQ -  multi map: 0x20001^M
[    0.265613] >>>> MAP for cpu: 1 | map: 2^M

@ Gerwin Klein Can you help me with this problem?

polijh commented 2 years ago

@bennoleslie @stevebob @jashank @ajaysusarla @lsf37 Can you help me with this problem? thanks!

lsf37 commented 2 years ago

Sorry can't help you with that one, I'm no expert on VMs. @Ivan-Velickovic can you help?

@polijh when you add commands or machine output to an issue, please surround the input/output by ``` and ``` (see GitHub markdown syntax), because GitHub rendering gets confused otherwise which makes the issue very hard to read. Example:

``` some output # another line ```

will show up as 

some output
# another line

I've added these to your post, so nothing to do for now, just for future issues.

polijh commented 2 years ago

Sorry can't help you with that one, I'm no expert on VMs. @Ivan-Velickovic can you help?

@polijh when you add commands or machine output to an issue, please surround the input/output by and (see GitHub markdown syntax), because GitHub rendering gets confused otherwise which makes the issue very hard to read. Example:

some output # another line

will show up as

some output
# another line

I've added these to your post, so nothing to do for now, just for future issues.

Thanks for your reply, I solved the problem by replacing a self-compiled linux kernel and the previous modification on psci node. It seems that the kernel Image provided by this repo has some problems or some debug prints are not removed. And the processing of the psci node by the function generate_psci_node caused some problems