Closed dhand-galois closed 4 years ago
dhand-galois
commented
4 years ago My first test involved adding the SiFive UART driver and creating the chosen entry w/ stdout-path in the device tree. But there was no output at all from booting the FreeBSD image on a Bluespec P2 FireSim AFI.
jrtc27
commented
4 years ago I assume you get output from bbl? What is your full device tree? When you say "adding the SiFive UART driver" do you mean you tried to compile in the driver to FreeBSD, and if so, how? FreeBSD should just be using SBI calls to bbl to print output though until a bit later in boot when it's had a chance to fully initialise, I believe.
dhand-galois
commented
4 years ago I did not get output from bbl, even though that is working on my Linux image (the bbl loader message). So that was going to be the first place I looked if I got back to debugging this. Perhaps the bbl contents did not fully rebuild after I ported the fix to make printing work there...
On the UART driver question, I:
riscv/sifive/sifive_uart.c optional uart_sifive fdt to src/conf/files.riscvdevice uart_sifive # sifive UART driver to sys/riscv/conf/GENERICsys/riscv/sifive/sifive_uart.c to remove it's dependence on dev/extres/clk/clk.h and set the divisor register with a pre-determined number.On the last point, I could not get it to build with the clocking dependency (not very familiar with FreeBSD's build system) and it likely would have computed the wrong divisor value from the device tree anyway, so I did that as a quick work around.
Device tree is:
/dts-v1/;
/ {
#address-cells = <1>;
#size-cells = <1>;
compatible = "freechips,rocketchip-unknown-dev";
model = "freechips,rocketchip-unknown";
L20: aliases {
serial0 = &L13;
};
L15: chosen {
bootargs = "earlyprintk console=hvc0 earlycon=sbi";
stdout-path = &L13;
};
L16: L19: cpus {
#address-cells = <1>;
#size-cells = <0>;
timebase-frequency = <50000000>;
L6: cpu@0 {
clock-frequency = <50000000>;
compatible = "galois,SSITH-bluespec_p2", "riscv";
d-cache-block-size = <64>;
d-cache-sets = <64>;
d-cache-size = <16384>;
d-tlb-sets = <1>;
d-tlb-size = <32>;
device_type = "cpu";
hardware-exec-breakpoint-count = <0>;
i-cache-block-size = <64>;
i-cache-sets = <64>;
i-cache-size = <16384>;
i-tlb-sets = <1>;
i-tlb-size = <32>;
mmu-type = "riscv,sv39";
next-level-cache = <&L7>;
reg = <0x0>;
riscv,isa = "rv64imafdc";
status = "okay";
timebase-frequency = <50000000>;
tlb-split;
L1: interrupt-controller {
#interrupt-cells = <1>;
compatible = "riscv,cpu-intc";
interrupt-controller;
};
};
};
L9: memory@80000000 {
device_type = "memory";
reg = <0x80000000 0x80000000>;
};
L18: soc {
#address-cells = <1>;
#size-cells = <1>;
compatible = "freechips,rocketchip-unknown-soc", "simple-bus";
ranges;
L12: blkdev-controller@40015000 {
compatible = "ucbbar,blkdev";
interrupt-parent = <&L4>;
interrupts = <1>;
reg = <0x40015000 0x1000>;
reg-names = "control";
};
L7: cache-controller@2010000 {
cache-block-size = <64>;
cache-level = <2>;
cache-sets = <1024>;
cache-size = <524288>;
cache-unified;
compatible = "sifive,inclusivecache0", "cache";
next-level-cache = <&L9>;
reg = <0x2010000 0x1000>;
reg-names = "control";
sifive,mshr-count = <7>;
};
L3: clint@10000000 {
compatible = "riscv,clint0";
interrupts-extended = <&L1 3 &L1 7>;
reg = <0x10000000 0x10000>;
reg-names = "control";
};
L2: debug-controller@0 {
compatible = "sifive,debug-013", "riscv,debug-013";
interrupts-extended = <&L1 65535>;
reg = <0x0 0x1000>;
reg-names = "control";
};
L0: error-device@3000 {
compatible = "sifive,error0";
reg = <0x3000 0x1000>;
};
L14: ice-nic@62100000 {
compatible = "ucbbar,ice-nic";
interrupt-parent = <&L4>;
interrupts = <3 4>;
reg = <0x62100000 0x1000>;
reg-names = "control";
};
L4: interrupt-controller@c000000 {
#interrupt-cells = <1>;
compatible = "riscv,plic0";
interrupt-controller;
interrupts-extended = <&L1 11 &L1 9>;
reg = <0xc000000 0x400000>;
reg-names = "control";
riscv,max-priority = <7>;
riscv,ndev = <16>;
};
L5: mmint@2000000 {
compatible = "ssith,mmint";
reg = <0x2000000 0x100>;
reg-names = "control";
};
L11: rom@70000000 {
compatible = "sifive,rom0";
reg = <0x70000000 0x10000>;
reg-names = "mem";
};
L13: serial@62300000 {
compatible = "sifive,uart0";
interrupt-parent = <&L4>;
interrupts = <2>;
reg = <0x62300000 0x1000>;
reg-names = "control";
};
};
};console=hvc0 is working in Linux where the actual SiFive UART driver is still not working (issue with it not determining the baud / clock properly). So the point on the SiFive UART driver is moot as you said - it should at least start booting and printing via calls to bbl.
jrtc27
commented
4 years ago If your bbl isn't printing anything then bbl has likely itself died and you may not even have reached FreeBSD's entry point, but it certainly then won't be able to print anything itself during early boot. Though stock bbl supports the sifive uart just fine, so if you tell it the right things in the device tree it really should work and not need to be patched from upstream. Then again the GFE bbl is using an outdated fork that is missing a bunch of the fixes done upstream to actually query the device tree, at least for the 16550 uart, so maybe the GFE one is broken but the upstream one works.
dhand-galois
commented
4 years ago Progress!
The issue was gfe/freebsd/Makefile did not apply the --with-mem-start=0xC0000000 argument while configuring bbl.
The issue now is that the first half of this log spat out in a couple seconds. The 2nd half of the log took over 5 minutes. Issue with the Timer/timebase-frequency setting?
Commencing simulation.
bbl loader
---<<BOOT>>---
KDB: debugger backends: ddb
KDB: current backend: ddb
Found 1 CPUs in the device tree
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-CURRENT #0 e75a79b70e3-c267832(HEAD)-dirty: Wed May 13 00:47:06 PDT 2020
dhand@besspin-fpga-3:/home/dhand/gfe/freebsd/obj/home/dhand/gfe/freebsd/cheribsd/riscv.riscv64/sys/TSFREEBSD-FPGA riscv
clang version 11.0.0 (https://github.com/llvm/llvm-project.git 49e20c4c9efe1c0e74f9c0dc224a8014b93faa3c)
WARNING: WITNESS option enabled, expect reduced performance.
Preloaded elf64 kernel "kernel" at 0xffffffc0049810c0.
SBI: Unknown (Legacy) Implementation
SBI Specification Version: 0.1
CPU(0): Unknown Implementer Unknown Processor
real memory = 2147483648 (2048 MB)
Physical memory chunk(s):
0x00000000c4ba3000 - 0x00000000fe5affff, 966840320 bytes (236045 pages)
avail memory = 961601536 (917 MB)
No static device mappings.
random: no preloaded entropy cache
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
VIMAGE (virtualized network stack) enabled
ULE: setup cpu 0
random: entropy device external interface
nfslock: pseudo-device
crypto: <crypto core>
null: <full device, null device, zero device>
openfirm: <Open Firmware control device>
WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 13.0.
mem: <memory>
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
plic0: <RISC-V PLIC> mem 0xc000000-0xc3fffff irq 3,4 on simplebus0
timer0: <RISC-V Timer>
Timecounter "RISC-V Timecounter" frequency 50000000 Hz quality 1000
Event timer "RISC-V Eventtimer" frequency 50000000 Hz quality 1000
rcons0: <RISC-V console>
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
cpu0: Nominal frequency 50Mhz
riscv64_cpu0: register <0>
simplebus0: <blkdev-controller@40015000> mem 0x40015000-0x40015fff irq 0 compat ucbbar,blkdev (no driver attached)
simplebus0: <cache-controller@2010000> mem 0x2010000-0x2010fff compat sifive,inclusivecache0 (no driver attached)
simplebus0: <error-device@3000> mem 0x3000-0x3fff compat sifive,error0 (no driver attached)
simplebus0: <ice-nic@62100000> mem 0x62100000-0x62100fff irq 1,2 compat ucbbar,ice-nic (no driver attached)
simplebus0: <mmint@2000000> mem 0x2000000-0x20000ff compat ssith,mmint (no driver attached)
simplebus0: <rom@70000000> mem 0x70000000-0x7000ffff compat sifive,rom0 (no driver attached)
uart0: <SiFive UART> mem 0x62300000-0x62300fff irq 5 on simplebus0
uart0: console (3682393,n,8,1)
uart0: fast interrupt
uart0: PPS capture mode: DCD
cryptosoft0: <software crypto>
crypto: assign cryptosoft0 driver id 0, flags 0x6000000
Device configuration finished.
procfs registered
Timecounters tick every 1.000 msec
md0: Embedded image 68157440 bytes at 0xffffffc000604180
lo0: bpf attached
vlan: initialized, using hash tables with chaining
tcp_init: net.inet.tcp.tcbhashsize auto tuned to 8192
IPsec: Initialized Security Association Processing.
Obsolete code will be removed soon: random(9) is the obsolete Park-Miller LCG from 1988
Trying to mount root from ufs:/dev/md0 []...
WARNING: WITNESS option enabled, expect reduced performance.
Warning: no time-of-day clock registered, system time will not be set accurately
start_init: trying /sbin/init
No suitable dump device was found.
Starting file system checks:
/dev/md0: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/md0: clean, 2351 free (7 frags, 293 blocks, 0.0% fragmentation)
/etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Setting hostuuid: fd728d0a-94ed-11ea-b193-e184d7171c70.
Setting hostid: 0x88e0c7e4.
Mounting local filesystems:.
mtree: /etc/mtree/BSD.sendmail.dist: No such file or directory
Setting hostname: gfe.
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: random: unblocking device.
.
lo0: link state changed to UP
Starting Network: lo0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Starting devd.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
devmatch: Can't read linker hints file.
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Updating motd:.
Mounting late filesystems:.
Updating /var/run/os-release done.
Starting cron.
devmatch: Can't read linker hints file.
Starting background file system checks in 60 seconds.
Wed May 13 07:58:52 UTC 2020
FreeBSD/riscv (gfe) (ttyu0)
login: root
May 13 07:59:09 gfe login[532]: ROOT LOGIN (root) ON ttyu0
FreeBSD 13.0-CURRENT (TSFREEBSD-FPGA) #0 e75a79b70e3-c267832(HEAD)-dirty: Wed May 13 00:47:06 PDT 2020
Welcome to FreeBSD!
Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums: https://forums.FreeBSD.org/
Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.
Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier
Edit /etc/motd.template to change this login announcement.
# sysctl hw.model hw.machine hw.ncpu
sysctl: unknown oid 'hw.model'
hw.machine: riscv
hw.ncpu: 1
# 
dmzimmerman
commented
4 years ago Where was the point where it stopped being fast and starting taking 5 minutes?
dhand-galois
commented
4 years ago Log with timestamps:
Commencing simulation.
[2020-05-14 01:52:24] bbl loader
[2020-05-14 01:52:24] ---<<BOOT>>---
[2020-05-14 01:52:24] KDB: debugger backends: ddb
[2020-05-14 01:52:24] KDB: current backend: ddb
[2020-05-14 01:52:24] Found 1 CPUs in the device tree
[2020-05-14 01:52:24] Copyright (c) 1992-2020 The FreeBSD Project.
[2020-05-14 01:52:24] Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
[2020-05-14 01:52:24] The Regents of the University of California. All rights reserved.
[2020-05-14 01:52:24] FreeBSD is a registered trademark of The FreeBSD Foundation.
[2020-05-14 01:52:24] FreeBSD 13.0-CURRENT #0 e75a79b70e3-c267832(HEAD)-dirty: Wed May 13 00:47:06 PDT 2020
[2020-05-14 01:52:24] dhand@besspin-fpga-3:/home/dhand/gfe/freebsd/obj/home/dhand/gfe/freebsd/cheribsd/riscv.riscv64/sys/TSFREEBSD-FPGA riscv
[2020-05-14 01:52:24] clang version 11.0.0 (https://github.com/llvm/llvm-project.git 49e20c4c9efe1c0e74f9c0dc224a8014b93faa3c)
[2020-05-14 01:52:25] WARNING: WITNESS option enabled, expect reduced performance.
[2020-05-14 01:52:29] Preloaded elf64 kernel "kernel" at 0xffffffc0049810c0.
[2020-05-14 01:52:29] SBI: Unknown (Legacy) Implementation
[2020-05-14 01:52:29] SBI Specification Version: 0.1
[2020-05-14 01:52:29] CPU(0): Unknown Implementer Unknown Processor
[2020-05-14 01:52:29] real memory = 2147483648 (2048 MB)
[2020-05-14 01:52:29] Physical memory chunk(s):
[2020-05-14 01:52:29] 0x00000000c4ba3000 - 0x00000000fe5affff, 966840320 bytes (236045 pages)
[2020-05-14 01:52:31] avail memory = 961601536 (917 MB)
[2020-05-14 01:52:31] No static device mappings.
[2020-05-14 01:52:33] random: no preloaded entropy cache
[2020-05-14 01:52:33] arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
[2020-05-14 01:52:33] VIMAGE (virtualized network stack) enabled
[2020-05-14 01:52:36] ULE: setup cpu 0
[2020-05-14 01:52:37] random: entropy device external interface
[2020-05-14 01:52:37] nfslock: pseudo-device
[2020-05-14 01:52:37] crypto: <crypto core>
[2020-05-14 01:52:37] null: <full device, null device, zero device>
[2020-05-14 01:52:37] openfirm: <Open Firmware control device>
[2020-05-14 01:52:37] WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 13.0.
[2020-05-14 01:52:37] mem: <memory>
[2020-05-14 01:52:37] ofwbus0: <Open Firmware Device Tree>
[2020-05-14 01:52:38] simplebus0: <Flattened device tree simple bus> on ofwbus0
[2020-05-14 01:52:38] plic0: <RISC-V PLIC> mem 0xc000000-0xc3fffff irq 3,4 on simplebus0
[2020-05-14 01:52:39] timer0: <RISC-V Timer>
[2020-05-14 01:52:39] Timecounter "RISC-V Timecounter" frequency 50000000 Hz quality 1000
[2020-05-14 01:52:39] Event timer "RISC-V Eventtimer" frequency 50000000 Hz quality 1000
[2020-05-14 01:52:39] rcons0: <RISC-V console>
[2020-05-14 01:52:39] cpulist0: <Open Firmware CPU Group> on ofwbus0
[2020-05-14 01:52:39] cpu0: <Open Firmware CPU> on cpulist0
[2020-05-14 01:52:39] cpu0: Nominal frequency 50Mhz
[2020-05-14 01:52:39] riscv64_cpu0: register <0>
[2020-05-14 01:52:40] simplebus0: <blkdev-controller@40015000> mem 0x40015000-0x40015fff irq 0 compat ucbbar,blkdev (no driver attached)
[2020-05-14 01:52:40] simplebus0: <cache-controller@2010000> mem 0x2010000-0x2010fff compat sifive,inclusivecache0 (no driver attached)
[2020-05-14 01:52:40] simplebus0: <error-device@3000> mem 0x3000-0x3fff compat sifive,error0 (no driver attached)
[2020-05-14 01:52:40] simplebus0: <ice-nic@62100000> mem 0x62100000-0x62100fff irq 1,2 compat ucbbar,ice-nic (no driver attached)
[2020-05-14 01:52:40] simplebus0: <mmint@2000000> mem 0x2000000-0x20000ff compat ssith,mmint (no driver attached)
[2020-05-14 01:52:40] simplebus0: <rom@70000000> mem 0x70000000-0x7000ffff compat sifive,rom0 (no driver attached)
[2020-05-14 01:52:40] uart0: <SiFive UART> mem 0x62300000-0x62300fff irq 5 on simplebus0
[2020-05-14 01:52:40] uart0: console (3682393,n,8,1)
[2020-05-14 01:52:40] uart0: fast interrupt
[2020-05-14 01:52:40] uart0: PPS capture mode: DCD
[2020-05-14 01:52:40] cryptosoft0: <software crypto>
[2020-05-14 01:52:40] crypto: assign cryptosoft0 driver id 0, flags 0x6000000
[2020-05-14 01:52:40] Device configuration finished.
[2020-05-14 01:52:41] procfs registered
[2020-05-14 01:52:41] Timecounters tick every 1.000 msec
[2020-05-14 01:52:42] md0: Embedded image 68157440 bytes at 0xffffffc000604180
[2020-05-14 01:52:42] lo0: bpf attached
[2020-05-14 01:52:42] vlan: initialized, using hash tables with chaining
[2020-05-14 01:52:42] tcp_init: net.inet.tcp.tcbhashsize auto tuned to 8192
[2020-05-14 01:52:44] IPsec: Initialized Security Association Processing.
[2020-05-14 01:52:44] Obsolete code will be removed soon: random(9) is the obsolete Park-Miller LCG from 1988
[2020-05-14 01:52:45] Trying to mount root from ufs:/dev/md0 []...
[2020-05-14 01:52:46] WARNING: WITNESS option enabled, expect reduced performance.
[2020-05-14 01:52:49] Warning: no time-of-day clock registered, system time will not be set accurately
[2020-05-14 01:52:49] start_init: trying /sbin/init
[2020-05-14 01:53:21] No suitable dump device was found.
[2020-05-14 01:53:34] Starting file system checks:
[2020-05-14 01:53:39] /dev/md0: FILE SYSTEM CLEAN; SKIPPING CHECKS
[2020-05-14 01:53:39] /dev/md0: clean, 2351 free (7 frags, 293 blocks, 0.0% fragmentation)
[2020-05-14 01:54:15] /etc/rc: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
[2020-05-14 01:54:24] Setting hostuuid: fd728d0a-94ed-11ea-b193-e184d7171c70.
[2020-05-14 01:54:25] Setting hostid: 0x88e0c7e4.
[2020-05-14 01:54:33] Mounting local filesystems:.
[2020-05-14 01:54:43] mtree: /etc/mtree/BSD.sendmail.dist: No such file or directory
[2020-05-14 01:55:21] Setting hostname: gfe.
[2020-05-14 01:56:06] Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
[2020-05-14 01:56:07] Feeding entropy: random: unblocking device.
[2020-05-14 01:56:20] .
[2020-05-14 01:57:23] lo0: link state changed to UP
[2020-05-14 01:58:11] Starting Network: lo0.
[2020-05-14 01:58:12] lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
[2020-05-14 01:58:12] options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
[2020-05-14 01:58:12] inet6 ::1 prefixlen 128
[2020-05-14 01:58:12] inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
[2020-05-14 01:58:12] inet 127.0.0.1 netmask 0xff000000
[2020-05-14 01:58:12] groups: lo
[2020-05-14 01:58:13] nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[2020-05-14 01:58:36] ELF ldconfig path: /lib /usr/lib /usr/lib/compat
[2020-05-14 01:58:41] Starting devd.
[2020-05-14 01:59:15] devmatch: Can't read linker hints file.
[2020-05-14 01:59:40] devmatch: Can't read linker hints file.
[2020-05-14 02:00:06] devmatch: Can't read linker hints file.
[2020-05-14 02:00:30] devmatch: Can't read linker hints file.
[2020-05-14 02:00:55] devmatch: Can't read linker hints file.
[2020-05-14 02:01:20] devmatch: Can't read linker hints file.
[2020-05-14 02:01:44] devmatch: Can't read linker hints file.
[2020-05-14 02:02:09] devmatch: Can't read linker hints file.
[2020-05-14 02:02:34] devmatch: Can't read linker hints file.
[2020-05-14 02:03:26] add host 127.0.0.1: gateway lo0 fib 0: route already in table
[2020-05-14 02:04:03] add host ::1: gateway lo0 fib 0: route already in table
[2020-05-14 02:04:06] add net fe80::: gateway ::1
[2020-05-14 02:04:08] add net ff02::: gateway ::1
[2020-05-14 02:04:11] add net ::ffff:0.0.0.0: gateway ::1
[2020-05-14 02:04:13] add net ::0.0.0.0: gateway ::1
[2020-05-14 02:05:07] Creating and/or trimming log files.
[2020-05-14 02:05:13] Starting syslogd.
[2020-05-14 02:07:34] Clearing /tmp (X related).
[2020-05-14 02:08:27] Updating motd:.
[2020-05-14 02:08:30] Mounting late filesystems:.
[2020-05-14 02:08:54] Updating /var/run/os-release done.
[2020-05-14 02:09:12] Starting cron.
[2020-05-14 02:10:32] devmatch: Can't read linker hints file.
[2020-05-14 02:10:41] Starting background file system checks in 60 seconds.
[2020-05-14 02:10:42]
[2020-05-14 02:10:45] Wed May 13 07:58:51 UTC 2020
[2020-05-14 02:10:53]
[2020-05-14 02:10:53] FreeBSD/riscv (gfe) (ttyu0)
[2020-05-14 02:10:53] Looks like the longest delays are around the networking messages, but could be a coincidence. There is a "feeding entropy" message in the middle of all those.
The messages after the timer initialization are still fairly quick so may not be an interrupt issue w/ that.
dhand-galois
commented
4 years ago After letting it boot up, top shows 30-40% runtime spent on servicing interrupts.
And vmstat output shows a very large number of interrupts shortly after boot, but the # of interrupts doesn't seem way off, I suppose. If the goal is to set it to 1 ms per interrupt, then this is showing a 10 minute period which sounds about right. Just not clear why so much CPU time is being consumed by these.
# vmstat -i
interrupt total rate
intc,5: 628305 764
plic0,2: uart0 1046 1
Total 629351 765I tried changing this line of sys/riscv/riscv/timer.c to increase the counter comparison value:
if (first != 0) {
//counts = ((uint32_t)et->et_frequency * first) >> 32;
counts = ((uint32_t)et->et_frequency * first) >> 30;
sbi_set_timer(get_cycles() + counts);
csr_set(sie, SIE_STIE);
return (0);
}That helped reduce the interrupt rate slightly but the boot process is still slow. However, timeout values specified in seconds will now be longer so it is just shifting the issue around.
I also booted the image on the VCU118 - turns out it's just about as slow there with the same high CPU consumption. This was using the develop branch of gfe, so this is not something specific to FireSim/CloudGFE. Are these expected behaviors?
jrtc27
commented
4 years ago You have Timecounters tick every 1.000 msec in your boot log, so yes, 764 interrupts per second is the right order of magnitude. Other than MIPS and 32-bit Arm, the default in sys/kern/subr_param.c is 1000 Hz, i.e. your 1ms. We use options HZ=100 in all our kernel configs, as a 1ms timer is far too short for a 100 MHz processor.
dhand-galois
commented
4 years ago That all makes sense - thanks.
I tried building a kernel with HZ=100 and things are operating a bit more smoothly. I booted up to login prompt in about 7 mins on a 50MHz build. Based on all this and the fact that it largely works the same as it does on the VCU118, I'm going to call this particular task "done". It's working on both the Chisel and Bluespec P2s in FireSim. We don't yet have driver support for the block or ethernet devices, but there's a separate issue tracking that.
dhand-galois
commented
4 years ago Another data point - I had a 90MHz build laying around. Now booting in about 3-4 mins there.
FreeBSD (built from gfe develop branch) is not currently booting on the Bluespec P2 using the FireSim environment.
This issue will track progress in getting this working.