Running 32bit linux on spike

[Zielasko]

Hi Im was trying to get a 32bit version of Linux running on spike for quite some time now.

Its quite hard to find much documentation on RISCV 32bit that works for the current version. In many cases available flags for configuration have changed (eg. --enable-32bit), so i got spike working for normal newlib programs, but not for any Linux kernel or the bbl.
I also tried building the 64bit toolchain and running 64bit Linux on spike, which worked to some extend.

Im using the master branch of all repositories, but i also tried other branches for some tools (and most available branches for riscv-linux).
I listed most of my build configurations below and what problems i have/had with them. It would be nice to get some help for what mistakes i might have made or some info if it is at all possible to get 32bit Linux running on spike with a current version of spike+toolchain.

32bit toolchain

first try using the build-rv32ima.sh script

build_project riscv-fesvr --prefix=$RISCV build_project riscv-isa-sim --prefix=$RISCV --with-fesvr=$RISCV --with-isa=rv32ima build_project riscv-gnu-toolchain --prefix=$RISCV --with-arch=rv32g --with-abi=ilp32d CC= CXX= build_project riscv-pk --prefix=$RISCV --host=riscv32-unknown-elf build_project riscv-openocd --prefix=$RISCV --enable-remote-bitbang --disable-werror

(i modified with-arch(ima->g) and with-abi(ilp32->ilp32d)

all projects build successfully. compiling a simple "hello world" program with riscv32-unknown-elf-gcc and executing it on spike results in:

riscv32-unknown-elf-gcc -o hello hello.c
spike pk hello
>Hello world!

but running bbl on spike doesnt print anything and gets stuck in a loop after some time (probably while trying to execute the dummy payload)

i tried to recompile spike and especially pk/bbl several times, but for rv32 it always gets stuck in a loop.

building without the default buildscript

toolchain

../configure --prefix=$RISCV --with-arch=rv32g --with-abi=ilp32d (same as build script)
make linux for linux-gnu-gcc

riscv-fesvr

../configure --prefix=$RISCV --target=riscv32-unknown-linux-gnu

spike

../configure --prefix=$RISCV --with-fesvr=$RISCV

riscv-pk

../configure --prefix=$RISCV --host=riscv32-unknown-linux-gnu

and the same for newlib ../configure --prefix=$RISCV --host=riscv32-unknown-elf

testing spike with pk

./spike --isa=rv32g pk
tell me what ELF to load!

riscv32-unknown-elf-gcc -o hello32 hello.c 

spike --isa=rv32g pk hello32
Hello World!

running bbl

same loop/error:

for spike32bit bbl32bitnewlib with no payload (so it should run dummypayload):
spike bbl:
no output, spike hangs

core   0: 0xffffffff80002bb0 (0xfe079ee3) bnez    a5, pc - 4
core   0: 0xffffffff80002bac (0x00072783) lw      a5, 0(a4)
core   0: 0xffffffff80002bb0 (0xfe079ee3) bnez    a5, pc - 4
core   0: 0xffffffff80002bac (0x00072783) lw      a5, 0(a4)
core   0: 0xffffffff80002bb0 (0xfe079ee3) bnez    a5, pc - 4
core   0: 0xffffffff80002bac (0x00072783) lw      a5, 0(a4)
core   0: 0xffffffff80002bb0 (0xfe079ee3) bnez    a5, pc - 4

../configure --prefix=$RISCV --host=riscv32-unknown-linux-gnu --with-payload=RISCV/kernel/vmlinux32defwBB results in the same loop with no output.

64bit toolchain

using the build script builds a working toolchain able to run pk and bbl (with dummy payload)

configuration for riscv64-pk ../configure --prefix=$RISCV --host=riscv64-unknown-linux-gnu ../configure --prefix=$RISCV --host=riscv64-unknown-elf(build with default buildscript config) both have same output(successful executing dummypayload):

./riscv64-tools/bin/spike bbl

This is bbl's dummy_payload.  To boot a real kernel, reconfigure
bbl with the flag --with-payload=PATH, then rebuild bbl.

../configure --prefix=$RISCV --host=riscv32-unknown-linux-gnu --with-payload=RISCV/kernel/vmlinux64defwBB

spike64 bbl64gnulinux --withpayload=linux64-kernel-defconfig+busybox64gnulinux

executing spike bbl:

core   0: 0xffffffe0000905a6 (0x10500073) wfi (args unknown)
core   0: 0xffffffe0000905aa (0x10500073) wfi (args unknown)
core   0: 0xffffffe0000905ae (0x0000bfe5) j       pc - 8
core   0: 0xffffffe0000905a6 (0x10500073) wfi (args unknown)
core   0: 0xffffffe0000905aa (0x10500073) wfi (args unknown)
core   0: 0xffffffe0000905ae (0x0000bfe5) j       pc - 8
core   0: 0xffffffe0000905a6 (0x10500073) wfi (args unknown)
core   0: 0xffffffe0000905aa (0x10500073) wfi (args unknown)

(this one might have been caused by some leftover files from building the rv32 versions, though i tried to reset all source dirs to a clean state each time and rebuilding had the same result) built again on my laptop, but using tinyconfig for the linuxkernel (also running Ubuntu):

executing spike bbl:

OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
Linux version 4.15.0-rc5-00046-gb9719aa (jan@jan-ubuntu) (gcc version 7.2.0 (GCC)) #1 Fri May 4 17:21:31 CEST 2018
bootconsole [early0] enabled
Initial ramdisk at: 0x        (ptrval) (388600 bytes)
Zone ranges:
  DMA      [mem 0x0000000080200000-0x00000000ffffffff]
  Normal   empty
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000080200000-0x00000000ffffffff]
Initmem setup node 0 [mem 0x0000000080200000-0x00000000ffffffff]
elf_hwcap is 0x112d
Built 1 zonelists, mobility grouping on.  Total pages: 516615
Kernel command line: 
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Sorting __ex_table...
Memory: 2060296K/2095104K available (1338K kernel code, 134K rwdata, 351K rodata, 468K init, 717K bss, 34808K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
riscv,cpu_intc,0: 64l local interrupts mapped
clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
Console: colour dummy device 80x25
console [tty0] enabled
bootconsole [early0] disabled

this time it prints some output, but after disabling early0, its just running the same code repeatedly:

core   0: 0x000000008000016c (0x09813983) ld      s3, 152(sp)
core   0: 0x0000000080000170 (0x0a013a03) ld      s4, 160(sp)
core   0: 0x0000000080000174 (0x0a813a83) ld      s5, 168(sp)
core   0: 0x0000000080000178 (0x0b013b03) ld      s6, 176(sp)
core   0: 0x000000008000017c (0x0b813b83) ld      s7, 184(sp) 
core   0: 0x0000000080000180 (0x0c013c03) ld      s8, 192(sp) 
core   0: 0x0000000080000184 (0x0c813c83) ld      s9, 200(sp) 
core   0: 0x0000000080000188 (0x0d013d03) ld      s10, 208(sp)
core   0: 0x000000008000018c (0x0d813d83) ld      s11, 216(sp)
core   0: 0x0000000080000190 (0x0e013e03) ld      t3, 224(sp) 
core   0: 0x0000000080000194 (0x0e813e83) ld      t4, 232(sp) 
core   0: 0x0000000080000198 (0x0f013f03) ld      t5, 240(sp)
core   0: 0x000000008000019c (0x0f813f83) ld      t6, 248(sp) 
core   0: 0x00000000800001a0 (0x01013103) ld      sp, 16(sp)
core   0: 0x00000000800001a4 (0x30200073) mret 
core   0: 0xffffffe0001c0142 (0x00008f95) sub     a5, a5, a3
core   0: 0xffffffe0001c0144 (0xfee7ebe3) bltu    a5, a4, pc - 10
core   0: 0xffffffe0001c013a (0x0207c7b3) div     a5, a5, zero 
core   0: 0xffffffe0001c013e (0xc01027f3) csrr    a5, time
core   0: exception trap_illegal_instruction, epc 0xffffffe0001c013e
core   0:           tval 0x0000000000000000
core   0: 0x0000000080000004 (0x34011173) csrrw   sp, mscratch, sp
core   0: 0x0000000080000008 (0x1a010063) beqz    sp, pc + 416
core   0: 0x000000008000000c (0x04a13823) sd      a0, 80(sp)

Compiling a Kernel

Following the guide from riscv-tools
make mrproper
make ARCH=riscv defconfig
and
make ARCH=riscv tinyconfig
setting crosscompiler to riscv-unknown-linux-gnu
(for both 32 and 64 version)
make ARCH=riscv -j8

this works for 64bit and 32bit, but for 32bit i had to remove all graphics DRM options from the (def).config file for some (non master) branches (i tested several), else it cant find some function:__ucmpdi2

with busybox

make allnoconfig set additional flags with menuconfig (cant find Support --install so i set it manually) this builds for 32 and 64bit

further creating directories as in the guide create inittabfile (link is broken)

ln -s /bin/busybox sbin/init
ln -s sbin/init init

(first path seems slightly wrong in the guide)

sudo mknod dev/console c 5 1
find . | cpio --quiet -o -H newc > riscv-linux/rootfs.cpio

add initframfs source file to kernel config building kernel succeeds witht he same config as above for 32 and 64

rebuilding riscv-pk and executing bbl gives the output listed above.

[jim-wilson]

32-bit linux support is known to be broken. There is no schedule for fixing it.

There are a number of problems here. One is the fact that the 32-bit glibc support has not been upstreamed, so 32-bit linux support is only available in the old and obsolete glibc version in riscv-gnu-toolchain. This old glibc version does not work with current qemu. It only works with the old qemu in riscv-gnu-toolchain. It is believed that there was a struct stat change somewhere along the way which isn't reflected in this old glibc tree. Building a 32-bit linux toolchain and running programs on old user-mode qemu mostly works, but there was a glibc ABI change for instruction cache flushing that is missing, and it is believed that the signal handler frame changed (probably a ucontext_t struct change) which is also missing. Any programs that needs either of these two features will fail. Another problem is that the kernel is apparently broken and/or qemu support is broken. The core linux kernel port was upstreamed last fall, but only the 64-bit support has been tested upstream.

There has been some discussion of issues in a thread on the riscv.org sw-dev mailing list. You could try asking there. This thread for instance https://groups.google.com/a/groups.riscv.org/forum/#!searchin/sw-dev/Qemu%7Csort:date/sw-dev/9mZDHPEiqds/Vs52Zb6yAAAJ has info about 32-bit linux builds, and has a proposed 32-bit kernel patch near the bottom.

In general, I can't recommend riscv-tools. It is hard to build and poorly maintained. I would suggest riscv-gnu-toolchain if you just want a toolchain, and freedom-u-sdk if you want to build kernel/buildroot and boot it on a simulator. riscv-tools might be useful if you want to learn how to build a system from scratch, and it requires doing a lot of stuff by hand.

[Zielasko]

Thanks. I tried building the freedom-u-sdk with make sim, but this fails for me, while trying to build vmlinux. Everything else compiles without any error, but for linux it can find freedom-u-sdk/linux/scripts/extract-cert.c:21:25: fatal error: openssl/bio.h File or directory not found.
I hope its okay to ask this here since riscv-tools is not exactly freedom-u-sdk.

Also about the riscv-tools 64bit linux i tried to build. Im not exactly sure what output to expect for bbl + tinyconfig + busybox. As i posted above, i get some basic output printed on the early0 bootconsole, which tells me at least something seems to work, but after that it stops printing anything. So if everything would be working as it should, i would be getting a login ash ?

[jim-wilson]

The kernel build failed while building a utility for the host. You need to install the missing package on the host, probably something like libssl-dev.

The default serial driver no longer works with spike, the simulator used with "make sim". "make qemu" should work, it gives you qemu instead of spike, qemu has more features than spike. If you really want to use spike, then you have to modify the kernel config file to enable the old serial driver. This has been mentioned in freedom-u-sdk issues. See for instance https://github.com/sifive/freedom-u-sdk/issues/48

[Zielasko]

Thanks again.
Since i needed libssl-dev before, i was pretty sure i had it installed. Installing this and switching to the branch hifive-unleashed, as suggested in the issue you linked, lets me successfully boot 64bit Linux on spike.