keystone-enclave / keystone

Keystone Enclave (QEMU + HiFive Unleashed)
Other
473 stars 135 forks source link

Overlapping Memory Regions; Boot Hangs #377

Open seceng-jan opened 1 year ago

seceng-jan commented 1 year ago

Hi,

i am trying to run keystone on an Alveo 250 FPGA using Chipyard and Firesim. When I execute the workload, the boot hangs after [ 0.249572] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled similar to #235. Firesim is in version 1.17.0 and FireMarshal is on the current master. I have tried several versions of FireMarshal, including the dev branch as suggested in the documentation but it didn't help.

I guess that the problem has to do with the memory layout. When I run the workload in qemu (./marshal -v launch boards/chipyard/firemarshal-2/keystone.json), it fails with the following message

List of screen session identifers:
keystone: keystone

Script started, output log file is '/home/jan/firesim/target-design/chipyard/software/firemarshal/runOutput/keystone-launch-2023-10-31--14-11-33-JLRGBJOV5MJ8HALZ/keystone/uartlog'.
qemu-system-riscv64: Some ROM regions are overlapping
These ROM regions might have been loaded by direct user request or by default.
They could be BIOS/firmware images, a guest kernel, initrd or some other file loaded into guest memory.
Check whether you intended to load all this guest code, and whether it has been built to load to the correct addresses.

The following two regions overlap (in the memory address space):
  /home/jan/firesim/target-design/chipyard/software/firemarshal/boards/chipyard/firemarshal-2/keystone/build/bootrom.build/bootrom.bin (addresses 0x0000000080000000 - 0x000000008000cf8d)
  /home/jan/firesim/target-design/chipyard/software/firemarshal/images/firechip/keystone/keystone-bin ELF program header segment 1 (addresses 0x0000000080000000 - 0x0000000080058710)
Script done.

Do you know where it's going wrong and what I can do to fix it?

This is the entire boot log on the FPGA:

Commencing simulation.

OpenSBI v1.1
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

[SM] Initializing ... hart [0]
[SM] Keystone security monitor has been initialized!
00002020004020800000000000002024000010400000000000000090600004000000000000000000000000000000000000000000000040000000000000040001
Platform Name             : ucb-bar,chipyard
Platform Features         : medeleg
Platform HART Count       : 1
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 1000000Hz
Platform Console Device   : sifive_uart
Platform HSM Device       : ---
Platform Reboot Device    : htif
Platform Shutdown Device  : htif
Firmware Base             : 0x80000000
Firmware Size             : 364 KB
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 0
Domain0 HARTs             : 0*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000080000000-0x000000008007ffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address      : 0x0000000080200000
Domain0 Next Arg1         : 0x0000000081200000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes

Boot HART ID              : 0
Boot HART Domain          : root
Boot HART Priv Version    : v1.11
Boot HART Base ISA        : rv64imafdcx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 4
Boot HART PMP Address Bits: 33
Boot HART MHPM Count      : 0
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109
[    0.000000] Linux version 5.19.0-dirty (jan@chipyard) (riscv64-unknown-linux-gnu-gcc (g2ee5e430018) 12.2.0, GNU ld (GNU Binutils) 2.39) #13 SMP Tue Oct 31 14:04
:46 UTC 2023
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Machine model: ucb-bar,chipyard
[    0.000000] No DTB passed to the kernel
[    0.000000] Forcing kernel command line to: console=hvc0 earlycon cma=1024M@0-4G initcall_debug
[    0.000000] earlycon: sifive0 at MMIO 0x0000000054000000 (options '')
[    0.000000] printk: bootconsole [sifive0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 1024 MiB at 0x00000000c0000000
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000047fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000047fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000047fffffff]
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10001
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI HSM extension detected
[    0.000000] riscv: base ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 18 pages/cpu s34104 r8192 d31432 u73728
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 4136455
[    0.000000] Kernel command line: console=hvc0 earlycon cma=1024M@0-4G initcall_debug
[    0.000000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000bc000000-0x00000000c0000000] (64MB)
[    0.000000] Virtual kernel memory layout:
[    0.000000]       fixmap : 0xffffffc6fee00000 - 0xffffffc6ff000000   (2048 kB)
[    0.000000]       pci io : 0xffffffc6ff000000 - 0xffffffc700000000   (  16 MB)
[    0.000000]      vmemmap : 0xffffffc700000000 - 0xffffffc800000000   (4096 MB)
[    0.000000]      vmalloc : 0xffffffc800000000 - 0xffffffd800000000   (  64 GB)
[    0.000000]       lowmem : 0xffffffd800000000 - 0xffffffdbffe00000   (  15 GB)
[    0.000000]       kernel : 0xffffffff80000000 - 0xffffffffffffffff   (2047 MB)
[    0.000000] Memory: 15389776K/16775168K available (5513K kernel code, 4823K rwdata, 2048K rodata, 2158K init, 383K bss, 336816K reserved, 1048576K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=1.
[    0.000000] rcu:     RCU debug extended QS entry/exit.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@c000000: mapped 4 interrupts with 1 handlers for 2 contexts.
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[    0.000003] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[    0.000473] Console: colour dummy device 80x25
[    0.000535] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)
[    0.000712] pid_max: default: 32768 minimum: 301
[    0.001413] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.001783] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.004020] cblist_init_generic: Setting adjustable number of callback queues.
[    0.004043] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.004247] riscv: ELF compat mode failed
[    0.004322] ASID allocator disabled (0 bits)
[    0.004616] rcu: Hierarchical SRCU implementation.
[    0.004628] rcu:     Max phase no-delay instances is 1000.
[    0.005363] EFI services will not be available.
[    0.005872] smp: Bringing up secondary CPUs ...
[    0.005888] smp: Brought up 1 node, 1 CPU
[    0.008839] devtmpfs: initialized
[    0.010636] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.010699] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.068611] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.069568] cpuidle: using governor menu
[    0.080095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.080113] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.082565] clocksource: Switched to clocksource riscv_clocksource
[    0.096266] NET: Registered PF_INET protocol family
[    0.099238] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.141909] tcp_listen_portaddr_hash hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    0.142499] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.142598] TCP established hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.145300] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    0.149846] TCP: Hash tables configured (established 131072 bind 65536)
[    0.150163] UDP hash table entries: 8192 (order: 7, 786432 bytes, linear)
[    0.151824] UDP-Lite hash table entries: 8192 (order: 7, 786432 bytes, linear)
[    0.153686] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.154532] RPC: Registered named UNIX socket transport module.
[    0.154579] RPC: Registered udp transport module.
[    0.154594] RPC: Registered tcp transport module.
[    0.154606] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.156583] workingset: timestamp_bits=62 max_order=22 bucket_order=0
[    0.167005] NFS: Registering the id_resolver key type
[    0.167117] Key type id_resolver registered
[    0.167133] Key type id_legacy registered
[    0.167289] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.167334] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.167690] 9p: Installing v9fs 9p2000 file system support
[    0.168295] NET: Registered PF_ALG protocol family
[    0.168463] io scheduler mq-deadline registered
[    0.168507] io scheduler kyber registered
[    0.249572] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
seceng-jan commented 1 year ago

Quick update: I was able to fix the overlapping memory regions. The problem was in the keystone.json file from the firemarshal-keystone git. I deleted the qemu-args line, then it works. The host-init always checks out the v1.0.0 branch of keystone, I modified that to use the dev branch. This is the modified host-init.c

#!/bin/sh

set -e

if [ $(command -v cmake3) ]; then
    echo "Using cmake3"
    alias cmake=cmake3
fi

if [ ! -d keystone ]; then
    git clone https://github.com/keystone-enclave/keystone keystone
fi

# set qemu -bios argument to bootrom.bin we will build
PWD=$(pwd)
sed -e 's#\-bios .*bootrom.bin#\-bios '$PWD'/keystone/build/bootrom.build/bootrom.bin#g' -i keystone.json

cd keystone
git checkout dev
./fast-setup.sh
. ./source.sh
mkdir -p build
cd build
cmake .. -Dfiresim=y
make patch
make bootrom
make qemu
make examples

The boot sequence in qemu still fails, however, at lest there is an error now. I am assuming that the same happens on the FPGA but without the error message.

[    0.203991] syscon-poweroff: probe of soc:poweroff failed with error -16
[    0.206469] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.223567] /dev/root: Can't open blockdev
[    0.224347] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.224546] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.19.0 #2
[    0.224699] Hardware name: riscv-virtio,qemu (DT)
[    0.224836] Call Trace:
[    0.224982] [<ffffffff80005238>] dump_backtrace+0x1c/0x24
[    0.225075] [<ffffffff803556d0>] show_stack+0x2c/0x38
[    0.225128] [<ffffffff8035bc90>] dump_stack_lvl+0x3c/0x54
[    0.225208] [<ffffffff8035bcbc>] dump_stack+0x14/0x1c
[    0.225287] [<ffffffff80355a6a>] panic+0x102/0x298
[    0.225372] [<ffffffff804012c0>] mount_block_root+0x15e/0x20a
[    0.225462] [<ffffffff804014ae>] mount_root+0x142/0x15c
[    0.225545] [<ffffffff80401600>] prepare_namespace+0x138/0x17a
[    0.225630] [<ffffffff80400ebe>] kernel_init_freeable+0x1e8/0x202
[    0.225721] [<ffffffff8035d75c>] kernel_init+0x1e/0x10a
[    0.225802] [<ffffffff80003310>] ret_from_exception+0x0/0xc
[    0.226224] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---