Use unmodified OpenSBI

[avpatel]

Using OpenSBI generic platform we can now use same OpenSBI firmware binaries on Dromajo which work on QEMU, Spike, and SiFive Unleashed board.

This series does few enhancements to load opensbi firmware, kernel and initrd images separately.

[renau]

Very nice patch. I tried and everything looks fine. There are some minor changes in the doc/Usage.md that would fix some minor issue.

The new boot.cfg has a Image and rootfs.cpio, but the Usage.md does not include the command to get them:

diff --git a/doc/setup.md b/doc/setup.md index b6af414..922ea5d 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -56,6 +56,8 @@ cp opensbi/build/platform/generic/firmware/fw_jump.bin .

To boot Linux (login:root password:root)

+cp linux-5.5/arch/riscv/boot/Image .
+cp buildroot-2019.08.1/output/images/rootfs.cpio .
 ../src/dromajo boot.cfg

A minor thing is that the kernel that your image includes is a linux-5.7.0-rc3 while the Usage.md has a linux-5.5 kernel. I'll be fine if you update the kernel to 5.6.0 or just cut&paste the result of running the current setup.md instructions.

If you do not want to rerun, this is the one that I got from reproducing the instructions:

--- a/doc/setup.md +++ b/doc/setup.md @@ -56,6 +56,8 @@ cp opensbi/build/platform/generic/firmware/fw_jump.bin .

To boot Linux (login:root password:root)

+cp linux-5.5/arch/riscv/boot/Image .
+cp buildroot-2019.08.1/output/images/rootfs.cpio .
 ../src/dromajo boot.cfg

@@ -118,7 +120,7 @@ To continue booting Linux:

 ../src/dromajo --load ck1 ./boot.cfg
 [    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
-[    0.000000] Linux version 5.7.0-rc3-00024-gb137564bb141 (anup@anup-ubuntu64) (gcc version 9.2.0 (GCC), GNU ld (GNU Binutils) 2.32) #2 SMP Tue Apr 28 12:37:40 IST 2020
+[    0.000000] Linux version 5.5.0 (renau@mada4) (gcc version 9.2.1 20191130 (Debian 9.2.1-21)) #1 SMP Thu May 7 19:50:22 PDT 2020
 [    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
 [    0.000000] printk: bootconsole [sbi0] enabled
 [    0.000000] Initial ramdisk at: 0x(____ptrval____) (4997120 bytes)
@@ -129,26 +131,19 @@ To continue booting Linux:
 [    0.000000] Early memory node ranges
 [    0.000000]   node   0: [mem 0x0000000080200000-0x00000000bfffffff]
 [    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000000bfffffff]
-[    0.000000] software IO TLB: mapped [mem 0xbad39000-0xbed39000] (64MB)
-[    0.000000] SBI specification v0.2 detected
-[    0.000000] SBI implementation ID=0x1 Version=0x7
-[    0.000000] SBI v0.2 TIME extension detected
-[    0.000000] SBI v0.2 IPI extension detected
-[    0.000000] SBI v0.2 RFENCE extension detected
-[    0.000000] SBI v0.2 HSM extension detected
-[    0.000000] riscv: ISA extensions acdfimsu
-[    0.000000] riscv: ELF capabilities acdfim
-[    0.000000] percpu: Embedded 17 pages/cpu s31640 r8192 d29800 u69632
+[    0.000000] software IO TLB: mapped [mem 0xbad3a000-0xbed3a000] (64MB)
+[    0.000000] elf_hwcap is 0x112d
+[    0.000000] percpu: Embedded 13 pages/cpu s23968 r0 d29280 u53248
 [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258055
 [    0.000000] Kernel command line: root=/dev/ram rw earlycon=sbi console=hvc0
 [    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
 [    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
 [    0.000000] Sorting __ex_table...
 [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
-[    0.000000] Memory: 942668K/1046528K available (6395K kernel code, 4229K rwdata, 4096K rodata, 226K init, 324K bss, 103860K reserved, 0K cma-reserved)
+[    0.000000] Memory: 954612K/1046528K available (1844K kernel code, 138K rwdata, 510K rodata, 2455K init, 216K bss, 91916K reserved, 0K cma-reserved)
 [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
 [    0.000000] rcu: Hierarchical RCU implementation.
-[    0.000000] rcu:    RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
+[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
 [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
 [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
 [    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
@@ -156,90 +151,43 @@ To continue booting Linux:
 [    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
 [    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
 [    0.000024] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns
-[    0.000741] Console: colour dummy device 80x25
-[    0.000977] printk: console [hvc0] enabled
-[    0.000977] printk: console [hvc0] enabled
-[    0.001378] printk: bootconsole [sbi0] disabled
-[    0.001378] printk: bootconsole [sbi0] disabled
-[    0.001852] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000)
-[    0.002373] pid_max: default: 32768 minimum: 301
-[    0.003071] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
-[    0.003469] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
-[    0.007280] rcu: Hierarchical SRCU implementation.
-[    0.008372] smp: Bringing up secondary CPUs ...
-[    0.008643] smp: Brought up 1 node, 1 CPU
-[    0.009592] devtmpfs: initialized
-[    0.011719] random: get_random_u32 called from bucket_table_alloc.isra.0+0x4e/0x154 with crng_init=0
-[    0.012611] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
-[    0.013565] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
-[    0.014873] NET: Registered protocol family 16
-[    0.053505] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
-[    0.058282] vgaarb: loaded
-[    0.059316] SCSI subsystem initialized
-[    0.060377] usbcore: registered new interface driver usbfs
-[    0.060772] usbcore: registered new interface driver hub
-[    0.061135] usbcore: registered new device driver usb
-[    0.064085] clocksource: Switched to clocksource riscv_clocksource
-[    0.081119] NET: Registered protocol family 2
-[    0.083124] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
-[    0.083603] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
-[    0.084614] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
-[    0.085593] TCP: Hash tables configured (established 8192 bind 8192)
-[    0.086122] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
-[    0.086561] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
-[    0.087283] NET: Registered protocol family 1
-[    0.088468] RPC: Registered named UNIX socket transport module.
-[    0.088767] RPC: Registered udp transport module.
-[    0.089010] RPC: Registered tcp transport module.
-[    0.089254] RPC: Registered tcp NFSv4.1 backchannel transport module.
-[    0.089580] PCI: CLS 0 bytes, default 64
-[    0.090174] Unpacking initramfs...
-[    0.199788] Freeing initrd memory: 4880K
-[    0.201533] workingset: timestamp_bits=62 max_order=18 bucket_order=0
-[    0.231709] NFS: Registering the id_resolver key type
-[    0.232079] Key type id_resolver registered
-[    0.232297] Key type id_legacy registered
-[    0.232528] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
-[    0.233201] 9p: Installing v9fs 9p2000 file system support
-[    0.234378] NET: Registered protocol family 38
-[    0.234669] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
-[    0.235036] io scheduler mq-deadline registered
-[    0.235271] io scheduler kyber registered
-update_mip: hartid=0 mask=0 value=0
-[    0.336665] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
-[    0.339248] sifive-serial 54000000.uart: IRQ index 0 not found
-[    0.340860] [drm] radeon kernel modesetting enabled.
-[    0.360022] loop: module loaded
-[    0.362591] libphy: Fixed MDIO Bus: probed
-[    0.365061] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
-[    0.365355] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
-[    0.365898] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
-[    0.366225] ehci-pci: EHCI PCI platform driver
-[    0.366545] ehci-platform: EHCI generic platform driver
-[    0.366934] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
-[    0.367259] ohci-pci: OHCI PCI platform driver
-[    0.367579] ohci-platform: OHCI generic platform driver
-[    0.368622] usbcore: registered new interface driver uas
-[    0.369022] usbcore: registered new interface driver usb-storage
-[    0.369613] mousedev: PS/2 mouse device common for all mice
-[    0.371251] usbcore: registered new interface driver usbhid
-[    0.371531] usbhid: USB HID core driver
-[    0.374434] NET: Registered protocol family 10
-[    0.376397] Segment Routing with IPv6
-[    0.376794] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
-[    0.378626] NET: Registered protocol family 17
-[    0.379467] 9pnet: Installing 9P2000 support
-[    0.379802] Key type dns_resolver registered
-[    0.381237] sifive-serial 54000000.uart: IRQ index 0 not found
-[    0.381861] sifive-serial 54000000.uart: IRQ index 0 not found
-[    0.383518] Freeing unused kernel memory: 224K
-[    0.388128] Run /init as init process
+[    0.000520] printk: console [hvc0] enabled
+[    0.000520] printk: console [hvc0] enabled
+[    0.000915] printk: bootconsole [sbi0] disabled
+[    0.000915] printk: bootconsole [sbi0] disabled
+[    0.001378] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000)
+[    0.001893] pid_max: default: 32768 minimum: 301
+[    0.002525] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
+[    0.002919] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
+[    0.006155] rcu: Hierarchical SRCU implementation.
+[    0.006953] smp: Bringing up secondary CPUs ...
+[    0.007190] smp: Brought up 1 node, 1 CPU
+[    0.008017] devtmpfs: initialized
+[    0.008741] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
+[    0.009227] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
+[    0.009684] thermal_sys: Registered thermal governor 'step_wise'
+[    0.009699] thermal_sys: Registered thermal governor 'user_space'
+[    0.016981] vgaarb: loaded
+[    0.017674] clocksource: Switched to clocksource riscv_clocksource
+[    0.019619] PCI: CLS 0 bytes, default 64
+[    0.677488] Unpacking initramfs...
+[    0.802073] Freeing initrd memory: 4880K
+[    0.803100] workingset: timestamp_bits=62 max_order=18 bucket_order=0
+[    0.889016] io scheduler mq-deadline registered
+[    0.889247] io scheduler kyber registered
+[    0.955638] random: get_random_bytes called from init_oops_id+0x22/0x2c with crng_init=0
+[    0.960977] Freeing unused kernel memory: 2452K
+[    0.961207] This architecture does not have kernel memory protection.
+[    0.961524] Run /init as init process
+mount: mounting sysfs on /sys failed: No such device
 Starting syslogd: OK
 Starting klogd: OK
 Running sysctl: OK
-Initializing random number generator... [    0.597752] random: dd: uninitialized urandom read (512 bytes read)
+Initializing random number generator... [    1.147173] random: dd: uninitialized urandom read (512 bytes read)
 done.
-Starting network: OK
+Starting network: ip: socket: Function not implemented
+ip: socket: Function not implemented
+FAIL

 Welcome to Dromajo Buildroot
 buildroot login:

[renau]

As a said, nice patch. I am fine with merging it, but can you patch the doc/Usage.md (I can patch it afterwards if you do not have time).

[avpatel]

Okay, I will update setup.md like you suggested.

[avpatel]

BTW, I see that you are embedding rootfs.cpio in kernel image. We don't need that as well because with "initrd" JSON tag we can load rootfs.cpio separately.

[avpatel]

I forgot to mention that OpenSBI v0.7 (or higher) has SBI HSM support. This means going forward with OpenSBI v0.7 (or higher) we support ordered booting using SBI HSM and you will need Linux-5.7 or higher to bring-up secondary HARTs.

This is a hard switch from old random booting (where all HART jumped into Linux at once) which was prone to bugs.

I will update doc/setup.md assuming Linux-5.7-rc4 and in early June 2020 we will be releasing OpenSBI v0.8.

[avpatel]

I have updated this PR .

After OpenSBI v0.8 release (early June 2020), we can directly use pre-build OpenSBI generic platform binaries that are released as part of OpenSBI release.

I have noticed few other minor fixes in FDT generation which I will send separately.

Regards, Anup

[renau]

I agree with your comments. Mostly: -patch the setup.md to have the new Linux kernel 5.7 -Do not include the root in the kernel now that it can be separate -Maybe pick a specific openSBI version so that it does not checkout the latest (easier to reproduce)

Thanks

[avpatel]

The OpenSBI generic platform was recently merged and it will be available in upcoming OpenSBI v0.8 release (early June 2020).

For time being, we build OpenSBI from gitrepo. Once OpenSBI v0.8 is released, you can directly use prebuilt binaries from release.

Regards, Anup

[avpatel]

I have updated doc/setup.md to checkout particular SHA of OpenSBI so that steps are reproducible for everyone.

Regards, Anup

[renau]

Looks good to me. Once Tommy does a pass we should be able to merge. Thanks

[renau]

I tried the latest changes. Everything works when I apply this change: diff --git a/doc/setup.md b/doc/setup.md index 1e182ce..7f7b3ee 100644 --- a/doc/setup.md +++ b/doc/setup.md @@ -36,7 +36,7 @@ make -j16 -C buildroot-2019.08.1 Assumption: you have the riscv64-unknown=linux-gnu- (GlibC) toolchain.

-export CROSS_COMPILE=riscv64-unknown=linux-gnu-
+export CROSS_COMPILE=riscv64-linux-gnu-
 wget -nc https://github.com/torvalds/linux/archive/v5.7-rc4.tar.gz
 tar -xvf v5.7-rc4.tar.gz
 make -C linux-5.7-rc4 ARCH=riscv defconfig
@@ -46,7 +46,7 @@ make -C linux-5.7-rc4 ARCH=riscv -j16
 ### OpenSBI (~ 1 min)

-export CROSS_COMPILE=riscv64-unknown=linux-gnu- +export CROSS_COMPILE=riscv64-linux-gnu- git clone https://github.com/riscv/opensbi.git cd opensbi git checkout 7be75f519f7705367030258c4410d9ff9ea24a6f -b temp

[et-tommythorn]

Yes, that matches my experience too.

[et-tommythorn]

I made those minor changes and pushed it. Thanks!