search

litex-hub / linux-on-litex-vexriscv

Linux on LiteX-VexRiscv
BSD 2-Clause "Simplified" License
593 stars 175 forks source link

Reduce minimum memory requirement to 32MB #4

Closed enjoy-digital closed 5 years ago

enjoy-digital commented 5 years ago

Currently, the minimum requirement is 64MB. Reducing the minimum requirement would be interesting to run on less expensive boards: ULX3S, Minispartan6.

An initial test has been done with the following patch:

From 60cabf74713750d0f8aaae969e0eec430a39fa07 Mon Sep 17 00:00:00 2001
From: Florent Kermarrec <florent@enjoy-digital.fr>
Date: Tue, 7 May 2019 10:20:29 +0200
Subject: [PATCH] try to reduce minimum memory requirement to 32MB

---
 buildroot/board/litex_vexriscv/litex_vexriscv.dts | 6 +++---
 emulator/main.c                                   | 2 +-
 serialboot.json                                   | 4 ++--
 sim.py                                            | 6 +++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/buildroot/board/litex_vexriscv/litex_vexriscv.dts b/buildroot/board/litex_vexriscv/litex_vexriscv.dts
index ddbedcc..2cb2c21 100644
--- a/buildroot/board/litex_vexriscv/litex_vexriscv.dts
+++ b/buildroot/board/litex_vexriscv/litex_vexriscv.dts
@@ -7,9 +7,9 @@
    model = "sifive,hifive-unleashed-a00";

    chosen {
-       bootargs = "mem=128M@0x40000000 rootwait console=hvc0 root=/dev/ram0 init=/sbin/init swiotlb=32";
-       linux,initrd-start = <0xC2000000>;
-       linux,initrd-end = <0xC2800000>; // max 8MB ramdisk image
+       bootargs = "mem=32M@0x40000000 rootwait console=hvc0 root=/dev/ram0 init=/sbin/init swiotlb=32";
+       linux,initrd-start = <0xC0800000>;
+       linux,initrd-end   = <0xC1000000>; // max 8MB ramdisk image
    };

    cpus {
diff --git a/emulator/main.c b/emulator/main.c
index ed81fd7..9f67ce6 100644
--- a/emulator/main.c
+++ b/emulator/main.c
@@ -15,7 +15,7 @@
 #include "riscv.h"

 #define LINUX_IMAGE_BASE 0xC0000000
-#define LINUX_DTB_BASE   0xC3000000
+#define LINUX_DTB_BASE   0xC1000000

 #define max(a,b) \
   ({ __typeof__ (a) _a = (a); \
diff --git a/serialboot.json b/serialboot.json
index 039da48..1362b66 100644
--- a/serialboot.json
+++ b/serialboot.json
@@ -1,6 +1,6 @@
 {
    "binaries/Image":         "0xc0000000",
-   "binaries/rootfs.cpio":   "0xc2000000",
-   "binaries/rv32.dtb":      "0xc3000000",
+   "binaries/rootfs.cpio":   "0xc0800000",
+   "binaries/rv32.dtb":      "0xc1000000",
    "emulator/emulator.bin":  "0x20000000"
 }
diff --git a/sim.py b/sim.py
index af271ec..37f5cbb 100755
--- a/sim.py
+++ b/sim.py
@@ -78,11 +78,11 @@ class LinuxSoC(SoCCore):
             cpu_type="vexriscv", cpu_variant="linux",
             with_uart=False,
             integrated_rom_size=0x8000,
-            integrated_main_ram_size=0x08000000, # 128MB
+            integrated_main_ram_size=0x02000000, # 32MB
             integrated_main_ram_init=get_mem_data({
                 "binaries/Image":         "0x00000000",
-                "binaries/rootfs.cpio":   "0x02000000",
-                "binaries/rv32.dtb":      "0x03000000"
+                "binaries/rootfs.cpio":   "0x00800000",
+                "binaries/rv32.dtb":      "0x01000000"
                 }, "little"),
             **kwargs)
         self.cpu.use_external_variant("VexRiscv.v")
-- 
2.17.1

But the simulation is not going to user space:

[    0.000000] No DTB passed to the kernel
[    0.000000] Linux version 5.0.9 (florent@lab) (gcc version 8.3.0 (Buildroot 2019.05-git-00938-g75f9fcd0c9)) #1 Thu May 2 17:43:30 CEST 2019
[    0.000000] Initial ramdisk at: 0x(ptrval) (8388608 bytes)
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000c0000000-0x00000000c7ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000c7ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000c7ffffff]
[    0.000000] elf_hwcap is 0x1100
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line: mem=32M@0x40000000 rootwait console=hvc0 root=/dev/ram0 init=/sbin/init swiotlb=32
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 119052K/131072K available (1957K kernel code, 92K rwdata, 317K rodata, 104K init, 184K bss, 12020K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x114c1bade8, max_idle_ns: 440795203839 ns
[    0.000160] sched_clock: 64 bits at 75MHz, resolution 13ns, wraps every 2199023255546ns
[    0.001527] Console: colour dummy device 80x25
[    0.008146] printk: console [hvc0] enabled
[    0.009202] Calibrating delay loop (skipped), value calculated using timer frequency.. 150.00 BogoMIPS (lpj=300000)
[    0.009881] pid_max: default: 32768 minimum: 301
[    0.016382] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.016920] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.046119] devtmpfs: initialized
[    0.063933] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.064558] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.123433] clocksource: Switched to clocksource riscv_clocksource
[    0.259769] Unpacking initramfs...
[    1.996061] workingset: timestamp_bits=30 max_order=15 bucket_order=0
[    2.168953] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    2.169412] io scheduler mq-deadline registered
[    2.169769] io scheduler kyber registered
[    3.033070] random: get_random_bytes called from init_oops_id+0x4c/0x60 with crng_init=0
[   10.056734] OF: fdt: not creating '/sys/firmware/fdt': CRC check failed
[   10.062543] Freeing unused kernel memory: 104K
[   10.062867] This architecture does not have kernel memory protection.
[   10.063256] Run /init as init process
- /home/florent/dev/linux/linux-on-litex-vexriscv/build/gateware/dut.v:1036: Verilog $finish

When running on the ULX3S (32MB SDRAM), the behaviour is seen. Investigation needs to be done with the simulations, ULX3S will probably work after that.

Dolu1990 commented 5 years ago

Later i can give a try with that linux images on the VexRiscv verilator simulation.

enjoy-digital commented 5 years ago

@Dolu1990 thanks, i haven't investigated yet, the emulator code catches something since it stops the simulation.

daveshah1 commented 5 years ago

Make sure the memory size is set correctly in https://github.com/enjoy-digital/linux-on-litex-vexriscv/blob/master/buildroot/board/litex_vexriscv/litex_vexriscv.dts#L63 as well as the command line. The startup logs still refer to 128MB of RAM.

daveshah1 commented 5 years ago

You might also want to reduce swiotlb further (I think each unit takes up 512KB, and I don't think it's important for anything atm)

Dolu1990 commented 5 years ago

Another 128MB : https://github.com/enjoy-digital/linux-on-litex-vexriscv/blob/7ab77c974f8af07dfa13531e87d65d5f9cd5834d/buildroot/board/litex_vexriscv/litex_vexriscv.dts#L63

futaris commented 5 years ago

looks like rootfs is being loaded on top of Image.

And the cmdline args like daveshah1 mentioned

futaris commented 5 years ago

Kernel takes up ~2MB. initramfs about 8MB.

12MB reserved.

Should still have 10MB free.

Dolu1990 commented 5 years ago

Be carefull about mega pages. I'm not sure how the linux would behave if things aren't 4 MB aligned. At least have multiple of 4MB for each parts i would say in a first time.

daveshah1 commented 5 years ago

I think reducing swiotlb should reduce the amount of "reserved" memory. In the longer term, the rootfs could probably be on the SD card rather than using an initrd.

enjoy-digital commented 5 years ago

Thanks, fixing the remaining 128MB makes it work:

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|

 (c) Copyright 2012-2019 Enjoy-Digital
 (c) Copyright 2012-2015 M-Labs Ltd

 BIOS built on May  7 2019 11:17:11
 BIOS CRC passed (21f2eb48)

--============ SoC info ================--
CPU:       VexRiscv @ 1MHz
ROM:       32KB
SRAM:      4KB
MAIN-RAM:  32768KB

--========= Peripherals init ===========--

--========== Boot sequence =============--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x20000000
--============= Liftoff! ===============--
VexRiscv Machine Mode software built May  7 2019 11:15:54
--========== Booting Linux =============--
[    0.000000] No DTB passed to the kernel
[    0.000000] Linux version 5.0.9 (florent@lab) (gcc version 8.3.0 (Buildroot 2019.05-git-00938-g75f9fcd0c9)) #1 Thu May 2 17:43:30 CEST 2019
[    0.000000] Initial ramdisk at: 0x(ptrval) (8388608 bytes)
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] elf_hwcap is 0x1101
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8128
[    0.000000] Kernel command line: mem=32M@0x40000000 rootwait console=hvc0 root=/dev/ram0 init=/sbin/init swiotlb=32
[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 21588K/32768K available (1957K kernel code, 92K rwdata, 317K rodata, 104K init, 184K bss, 11180K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x114c1bade8, max_idle_ns: 440795203839 ns
[    0.000160] sched_clock: 64 bits at 75MHz, resolution 13ns, wraps every 2199023255546ns
[    0.001538] Console: colour dummy device 80x25
[    0.008144] printk: console [hvc0] enabled
[    0.009248] Calibrating delay loop (skipped), value calculated using timer frequency.. 150.00 BogoMIPS (lpj=300000)
[    0.009927] pid_max: default: 32768 minimum: 301
[    0.016569] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.017112] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.046477] devtmpfs: initialized
[    0.065334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.066348] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.128120] clocksource: Switched to clocksource riscv_clocksource
[    0.264355] Unpacking initramfs...
[    2.008315] workingset: timestamp_bits=30 max_order=13 bucket_order=0
[    2.181809] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    2.182267] io scheduler mq-deadline registered
[    2.182623] io scheduler kyber registered
[    3.044680] random: get_random_bytes called from init_oops_id+0x4c/0x60 with crng_init=0
[    3.057582] Freeing unused kernel memory: 104K
[    3.057905] This architecture does not have kernel memory protection.
[    3.058295] Run /init as init process
mount: mounting tmpfs on /dev/shm failed: Invalid argument
mount: mounting tmpfs on /tmp failed: Invalid argument
mount: mounting tmpfs on /run failed: Invalid argument
Starting syslogd: OK
Starting klogd: OK
Initializing random number generator... [    4.395597] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: ip: socket: Function not implemented
ip: socket: Function not implemented
FAIL

Welcome to Buildroot
buildroot login: root
login[48]: root login on 'hvc0'
#

This still needs to be merged correctly, but at leat we know it works.

daveshah1 commented 5 years ago

Board-specific device trees would probably be a good idea, so all boards can use their full amount of RAM. This would also be useful once we start adding other peripherals like Ethernet and SD cards.

enjoy-digital commented 5 years ago

@daveshah1 yes i was also thinking about that, or even generating the device tree directly from the target.

enjoy-digital commented 5 years ago

For now i added board-specific device trees and dts are recompiled before loading/flashing the board. The default dts/dtb is the 32MB variant which will run on all targets.