Hackaday Badge : optimize performance

[pdp7]

Related to comments in PR https://github.com/litex-hub/litex-boards/pull/31#issuecomment-573301387

The performance of the ECP5 Hackaday Badge with 32MB SDRAM is "painfully" slow.

@mithro suggested there could be some issue with the configuration.

@enjoy-digital has attempted some optimizations: https://github.com/litex-hub/litex-boards/pull/31#issuecomment-573050963

With enjoy-digital/litedram@34e6c24 and enjoy-digital/litex@fa22d6a we have a ~10% boot time speedup on designs using SDRAM:

• De0Nano: 94.6.s to 84.6s
• ULX3S: 75.9s to 68.2s

On Arty with DDR3 the gain is effect more limited: 8.7s to 8.4s. That would be interesting to test this on the badge.

I will measure if the boot time improve

[pdp7]

@enjoy-digital thanks, it seems that boot to login prompt is 48 seconds faster.

Dropbear SSH starts after 258 seconds on the optimized version versus 306 seconds in the original version.

Here is it getting to login:

I have uploaded asciinema: https://asciinema.org/a/uf4BHtlLQT0w6pZZFAzQ1uk26

versus the original asciinema before optimization: https://asciinema.org/a/292299

The original version login prompt:

[enjoy-digital]

Thanks for the results, so you also have the ~10% performance gain. I'll continue looking at the bottlenecks next week if possible.

[enjoy-digital]

@pdp7: https://github.com/enjoy-digital/litedram/commit/721c84bad0a14e16c7fd7b6ebe01ad43bff3bcf8 and https://github.com/enjoy-digital/litex/commit/39ce39a298f576da8f4ccc97873a3a78f03cc4fe should give interesting boot time speed up on the Hackaday badge :)

If you want to try, you can rebuild with upstream LiteX/LiteDRAM or just use the bitstream attached: hadbadge_2020_01_13.zip

[pdp7]

@enjoy-digital WOW! much faster! It gets to login in 28 seconds (previous version was 258 seconds).

Recording: https://asciinema.org/a/Pcm3vd1BEdEKY9srYX6MsNfCE

Text:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv$ lxterm --images=images.json /dev/ttyUSB0 --speed=1e6 --no-crc
[LXTERM] Starting....
lBIOS CRC passed (561ab1e2)

 Migen git sha1: 063188e
 LiteX git sha1: --------

--=============== SoC ==================--
CPU:       VexRiscv @ 48MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  32768KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 76Mbps Reads: 96Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading buildroot/Image to 0xc0000000 (4545524 bytes)...
[LXTERM] Upload complete (93.4KB/s).
[LXTERM] Uploading buildroot/rootfs.cpio to 0xc0800000 (8029184 bytes)...
[LXTERM] Upload complete (93.3KB/s).
[LXTERM] Uploading buildroot/rv32.dtb to 0xc1000000 (1587 bytes)...
[LXTERM] Upload complete (86.9KB/s).
[LXTERM] Uploading emulator/emulator.bin to 0x20000000 (9584 bytes)...
[LXTERM] Upload complete (92.8KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
KExecuting booted program at 0x20000000

--============= Liftoff! ===============--
VexRiscv Machine Mode software built Jan 13 2020 18:35:20
--========== Booting Linux =============--
[    0.000000] No DTB passed to the kernel
[    0.000000] Linux version 5.0.13 (florent@lab) (gcc version 8.3.0 (Buildroot 2019.08-rc2-00011-gad9efda578)) #1 Thu Sep 12 14:20:26 CEST 2019
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Initial ramdisk at: 0x(ptrval) (8388608 bytes)
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] elf_hwcap is 0x1101
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8128
[    0.000000] Kernel command line: mem=32M@0xc0000000 rootwait console=liteuart earlycon=sbi root=/dev/ram0 init=/sbin/init swiotlb=32
[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 19812K/32768K available (3415K kernel code, 148K rwdata, 509K rodata, 140K init, 216K bss, 12956K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xb11fd3bfb, max_idle_ns: 440795203732 ns
[    0.000529] sched_clock: 64 bits at 48MHz, resolution 20ns, wraps every 4398046511096ns
[    0.006044] Console: colour dummy device 80x25
[    0.009270] Calibrating delay loop (skipped), value calculated using timer frequency.. 96.00 BogoMIPS (lpj=192000)
[    0.012397] pid_max: default: 32768 minimum: 301
[    0.034378] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.037383] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.159441] devtmpfs: initialized
[    0.241518] random: get_random_bytes called from setup_net+0x4c/0x188 with crng_init=0
[    0.256506] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.259731] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.285449] NET: Registered protocol family 16
[    0.764742] clocksource: Switched to clocksource riscv_clocksource
[    1.526244] NET: Registered protocol family 2
[    1.568199] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes)
[    1.571916] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    1.575159] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    1.577957] TCP: Hash tables configured (established 1024 bind 1024)
[    1.587321] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.590491] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.635099] Unpacking initramfs...
[    7.896109] Initramfs unpacking failed: junk in compressed archive
[    7.953828] workingset: timestamp_bits=30 max_order=13 bucket_order=0
[    9.236479] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    9.239022] io scheduler mq-deadline registered
[    9.240118] io scheduler kyber registered
[   13.977936] f0001000.serial: ttyLXU0 at MMIO 0xf0001000 (irq = 0, base_baud = 0) is a liteuart
[   13.980311] printk: console [liteuart0] enabled
[   13.980311] printk: console [liteuart0] enabled
[   13.982986] printk: bootconsole [sbi0] disabled
[   13.982986] printk: bootconsole [sbi0] disabled
[   14.058793] libphy: Fixed MDIO Bus: probed
[   14.074974] i2c /dev entries driver
[   14.247476] NET: Registered protocol family 10
[   14.307990] Segment Routing with IPv6
[   14.315230] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   14.455690] Freeing unused kernel memory: 140K
[   14.457897] This architecture does not have kernel memory protection.
[   14.459159] Run /init as init process
mount: mounting tmpfs on /dev/shm failed: Invalid argument
mount: mounting tmpfs on /tmp failed: Invalid argument
mount: mounting tmpfs on /run failed: Invalid argument
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Initializing random number generator... [   23.056430] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: OK
Starting dropbear sshd: [   27.316271] random: dropbear: uninitialized urandom read (32 bytes read)
OK

Welcome to Buildroot
buildroot login: 

[enjoy-digital]

Great :) thanks for testing. That would also be interesting to test with with l2_data_width=256 here: https://github.com/enjoy-digital/litex/blob/master/litex/soc/integration/soc_sdram.py#L29

[pdp7]

@enjoy-digital thanks, that is even faster by about 3 seconds, so a 12% speed-up.

pdp7@x1:~/dev/enjoy/litex$ git diff
diff --git a/litex/soc/integration/soc_sdram.py b/litex/soc/integration/soc_sdram.py
index 2535e85d..29873a8d 100644
--- a/litex/soc/integration/soc_sdram.py
+++ b/litex/soc/integration/soc_sdram.py
@@ -26,7 +26,7 @@ class SoCSDRAM(SoCCore):
     }
     csr_map.update(SoCCore.csr_map)

-    def __init__(self, platform, clk_freq, l2_size=8192, l2_data_width=128, **kwargs):
+    def __init__(self, platform, clk_freq, l2_size=8192, l2_data_width=256, **kwargs):
         SoCCore.__init__(self, platform, clk_freq, **kwargs)
         if not self.integrated_main_ram_size:
             if self.cpu_type is not None and self.csr_data_width > 32:

boot log:

--============= Liftoff! ===============--
VexRiscv Machine Mode software built Jan 13 2020 20:15:20
--========== Booting Linux =============--
[    0.000000] No DTB passed to the kernel
[    0.000000] Linux version 5.0.13 (florent@lab) (gcc version 8.3.0 (Buildroot 2019.08-rc2-00011-gad9efda578)) #1 Thu Sep 12 14:20:26 CEST 2019
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Initial ramdisk at: 0x(ptrval) (8388608 bytes)
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000c1ffffff]
[    0.000000] elf_hwcap is 0x1101
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8128
[    0.000000] Kernel command line: mem=32M@0xc0000000 rootwait console=liteuart earlycon=sbi root=/dev/ram0 init=/sbin/init swiotlb=32
[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 19812K/32768K available (3415K kernel code, 148K rwdata, 509K rodata, 140K init, 216K bss, 12956K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xb11fd3bfb, max_idle_ns: 440795203732 ns
[    0.000493] sched_clock: 64 bits at 48MHz, resolution 20ns, wraps every 4398046511096ns
[    0.005687] Console: colour dummy device 80x25
[    0.008652] Calibrating delay loop (skipped), value calculated using timer frequency.. 96.00 BogoMIPS (lpj=192000)
[    0.010779] pid_max: default: 32768 minimum: 301
[    0.031834] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.033776] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.144371] devtmpfs: initialized
[    0.217687] random: get_random_bytes called from setup_net+0x4c/0x188 with crng_init=0
[    0.231539] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.234582] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.256763] NET: Registered protocol family 16
[    0.683956] clocksource: Switched to clocksource riscv_clocksource
[    1.355650] NET: Registered protocol family 2
[    1.394116] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes)
[    1.397573] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    1.399693] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    1.402417] TCP: Hash tables configured (established 1024 bind 1024)
[    1.410684] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.413647] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.453714] Unpacking initramfs...
[    6.885797] Initramfs unpacking failed: junk in compressed archive
[    6.935727] workingset: timestamp_bits=30 max_order=13 bucket_order=0
[    8.083266] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    8.085623] io scheduler mq-deadline registered
[    8.086685] io scheduler kyber registered
[   12.262322] f0001000.serial: ttyLXU0 at MMIO 0xf0001000 (irq = 0, base_baud = 0) is a liteuart
[   12.265527] printk: console [liteuart0] enabled
[   12.265527] printk: console [liteuart0] enabled
[   12.267071] printk: bootconsole [sbi0] disabled
[   12.267071] printk: bootconsole [sbi0] disabled
[   12.334421] libphy: Fixed MDIO Bus: probed
[   12.349111] i2c /dev entries driver
[   12.501259] NET: Registered protocol family 10
[   12.554775] Segment Routing with IPv6
[   12.561424] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[   12.685369] Freeing unused kernel memory: 140K
[   12.686329] This architecture does not have kernel memory protection.
[   12.687472] Run /init as init process
mount: mounting tmpfs on /dev/shm failed: Invalid argument
mount: mounting tmpfs on /tmp failed: Invalid argument
mount: mounting tmpfs on /run failed: Invalid argument
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Initializing random number generator... [   20.245308] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: OK
Starting dropbear sshd: [   24.034199] random: dropbear: uninitialized urandom read (32 bytes read)
OK

Welcome to Buildroot
buildroot login: root

[enjoy-digital]

@pdp7: thanks, l2_data_width=128 should be a good trade off between resource usage / performance so we can probably keep that for now.

[pdp7]

While the kernel boots in under 30 seconds, it currently it takes about 2 minutes to load the kernel Image and rootfs.cpio:

lxterm --images=images.json /dev/ttyUSB0 --speed=1e6 --no-crc

The badge and cartridge both have:

• Winbond W25Q128JVSIQ 128 Mbit NOR flash chip

@mithro suggested I try to use the built-in Flash to store those, rather than load over serial.

@enjoy-digital @xobs @gregdavill @jcreedon @mwelling: Do you know of board configs I can use as an example to store the kernel and rootfs in flash?

[enjoy-digital]

@pdp7 you can look at the Arty. It was working at some point but i'm not sure the linux binaries are still fitting in the SPI Flash. The other thing is that you now need to make sure to append length and CRC with https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/mkmscimg.py before writing the images. You can find the flash offsets here: https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/bios/boot.c#L411-L414

[enjoy-digital]

The first thing to do is to add the Flash pins to your board, similar to https://github.com/litex-hub/litex-boards/blob/master/litex_boards/official/platforms/versa_ecp5.py#L47-L51. Then add "spiflash" to the capabilities: https://github.com/litex-hub/linux-on-litex-vexriscv/blob/master/make.py#L181 and test the build.

On hardware, you can then check if the SPI Flash is working correctly with the mr/fe/fw commands of the BIOS: https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/bios/main.c#L345-L350

It's possible you will have to adjust: https://github.com/litex-hub/linux-on-litex-vexriscv/blob/master/soc_linux.py#L110

[pdp7]

@enjoy-digital thank for you for insights. I am looking at datasheet. and it looks like 8 dummy bytes:

and I'm trying this diff:

diff --git a/make.py b/make.py
index f63e469..44574c2 100755
--- a/make.py
+++ b/make.py
@@ -163,9 +163,12 @@ class ULX3S(Board):
 # HADBadge support ---------------------------------------------------------------------------------

 class HADBadge(Board):
+    SPIFLASH_PAGE_SIZE = 256
+    SPIFLASH_SECTOR_SIZE = 64*kB
     def __init__(self):
         from litex_boards.targets import hadbadge
-        Board.__init__(self, hadbadge.BaseSoC, {"serial"})
+        Board.__init__(self, hadbadge.BaseSoC, {"serial", "spiflash"})
+        #Board.__init__(self, arty.EthernetSoC, {"serial", "ethernet", "spiflash", "leds", "rgb_led", "switches", "spi", "i2c", "xadc", "icap_bit"})

     def load(self):
         os.system("dfu-util --alt 2 --download build/hadbadge/gateware/top.bit --reset")
diff --git a/soc_linux.py b/soc_linux.py
index 98f6d7e..c76f373 100644
--- a/soc_linux.py
+++ b/soc_linux.py
@@ -63,10 +63,10 @@ def SoCLinux(soc_cls, **kwargs):

         def add_spi_flash(self):
             # TODO: add spiflash1x support
-            spiflash_pads = self.platform.request("spiflash4x")
+            spiflash_pads = self.platform.request("spiflash")
             self.submodules.spiflash = SpiFlash(
                 spiflash_pads,
-                dummy=11,
+                dummy=8,
                 div=2,
                 with_bitbang=True,
                 endianness=self.cpu.endianness)

 BIOS built on Jan 30 2020 22:02:40
 BIOS CRC passed (33e2c366)

 Migen git sha1: 063188e
 LiteX git sha1: --------

--=============== SoC ==================--
CPU:       VexRiscv @ 48MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  32768KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 79Mbps Reads: 103Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Loading emulator.bin from flash...
Error: Invalid image length 0xffffffff
Booting from flash...
Error: Invalid image length 0xffffffff
No boot medium found

--============= Console ================--
litex> 

From soc_linux.py:

        soc_cls.mem_map = {
            "rom":          0x00000000,
            "sram":         0x10000000,
            "emulator_ram": 0x20000000,
            "ethmac":       0xb0000000,
            "spiflash":     0xd0000000,
            "main_ram":     0xc0000000,
            "csr":          0xf0000000,
        }

It looks like I can read the flash ok:

litex> mr 0xd0000000 256
Memory dump:
0xd0000000  ff 00 50 61 72 74 3a 20 4c 46 45 35 55 2d 34 35  ..Part: LFE5U-45
0xd0000010  46 2d 38 43 41 42 47 41 33 38 31 00 ff ff ff bd  F-8CABGA381.....
0xd0000020  b3 ff ff ff ff 3b 00 00 00 e2 00 00 00 41 11 20  .....;.......A. 
0xd0000030  43 22 00 00 00 40 00 00 00 46 00 00 00 82 91 24  C"...@...F.....$
0xd0000040  fe 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd0000050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd0000060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd0000070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd0000080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd0000090  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd00000a0  00 00 00 00 00 00 00 00 00 00 00 64 f8 ff 00 00  ...........d....
0xd00000b0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd00000c0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd00000d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd00000e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
0xd00000f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
litex> 

[pdp7]

you can look at the Arty. It was working at some point but i'm not sure the linux binaries are still fitting in the SPI Flash. The other thing is that you now need to make sure to append length and CRC with https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/mkmscimg.py before writing the images. You can find the flash offsets here: https://github.com/enjoy-

thanks, do you know what tool I would use to write the files to flash?

[enjoy-digital]

You can try to use lxterm with --flash: https://github.com/enjoy-digital/litex/pull/305

[pdp7]

You can try to use lxterm with --flash: enjoy-digital/litex#305

Thanks, I'm trying:

lxterm --images=images.json /dev/ttyUSB0 --speed=1e6 --no-crc --flash

and:

 BIOS built on Jan 30 2020 22:02:40
 BIOS CRC passed (33e2c366)

 Migen git sha1: 063188e
 LiteX git sha1: --------

--=============== SoC ==================--
CPU:       VexRiscv @ 48MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  32768KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 79Mbps Reads: 103Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Flashing buildroot/Image to 0xc0000000 (4545524 bytes)...
[LXTERM] Upload complete (89.9KB/s).
[LXTERM] Flashing buildroot/rootfs.cpio to 0xc0800000 (8029184 bytes)...
[LXTERM] Upload complete (89.9KB/s).
[LXTERM] Flashing buildroot/rv32.dtb to 0xc1000000 (1808 bytes)...
[LXTERM] Upload complete (84.9KB/s).
[LXTERM] Flashing emulator/emulator.bin to 0x20000000 (9584 bytes)...
[LXTERM] Upload complete (89.9KB/s).
[LXTERM] Done.
KCCCCToo many consecutive errors, abortingLoading emulator.bin from flash...
Error: Invalid image length 0xffffffff
Booting from flash...
Error: Invalid image length 0xffffffff
No boot medium found

--============= Console ================--
litex> "�# !#.1#,A#*Q#(a#&q#&���;�7e@�ED�07�@����7iK"�"'"��"�'�"�'�c"�'���"�

followed by more garbage output.

I think I must have something not right in the configuration.

[enjoy-digital]

As i was explaining in https://github.com/litex-hub/litex-boards/issues/35#issuecomment-580437036, the files will first need to be patched with https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/mkmscimg.py to add the CRC and length and be sure to use the same offsets than in https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/bios/boot.c#L411-L414. But you will also probably need to disable some Linux features to reduce the kernel/rootfs size depending how large your SPI Flash is.

[pdp7]

@enjoy-digital thank you, I ran the mkmscimg.py and created .fbi files:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv$ cat images.json.fbi 
{
    "buildroot/Image.fbi":        "0x00000000",
    "buildroot/rootfs.cpio.fbi":   "0x00400000",
    "buildroot/rv32.dtb.fbi":     "0x00B00000",
    "emulator/emulator.bin.fbi":  "0x00B01000"
}

Do you know if litex_term.py uses the address as an offset, or is 0x00000000 the actual address?

@mithro is thinking that would overwrite the gateware.

UPDATE: that does seem to be what is happening. The gateware disappears and I have to run python3 make.py --board=hadbadge --load to restore it.

[pdp7]

I see that:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv$ git grep FLASH_BOOT_ADDRESS
soc_linux.py:                self.add_constant("FLASH_BOOT_ADDRESS", 0x00400000)

so I added 0x00400000 to the addresses in images.json.fbi:

{
    "buildroot/Image.fbi":         "0x00400000",
    "buildroot/rootfs.cpio.fbi":   "0x00800000",
    "buildroot/rv32.dtb.fbi":      "0x00F00000",
    "emulator/emulator.bin.fbi":   "0x00F01000"
}

[pdp7]

I'm trying to get the Hackaday Badge to boot from the SPI flash as it takes 2 mintues to load the kernel and rootfs over serial.

Repo version:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/buildroot$ git log --oneline -1
331ddce (HEAD -> master, origin/master, origin/HEAD) Merge pull request #103 from antmicro/sdram-timings

Repo diff:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/buildroot$ git diff
diff --git a/make.py b/make.py
index e472c25..27dcdd9 100755
--- a/make.py
+++ b/make.py
@@ -179,8 +179,8 @@ class ULX3S(Board):
 # HADBadge support ---------------------------------------------------------------------------------

 class HADBadge(Board):
-    SPIFLASH_PAGE_SIZE    = 256   # CHECKME
-    SPIFLASH_SECTOR_SIZE  = 64*kB # CHECKME
+    SPIFLASH_PAGE_SIZE    = 256
+    SPIFLASH_SECTOR_SIZE  = 64*kB
     SPIFLASH_DUMMY_CYCLES = 8
     def __init__(self):
         from litex_boards.targets import hadbadge
diff --git a/soc_linux.py b/soc_linux.py
index ecf875b..df90848 100644
--- a/soc_linux.py
+++ b/soc_linux.py
@@ -104,7 +104,8 @@ def SoCLinux(soc_cls, **kwargs):

         def add_spi_flash(self, dummy_cycles):
             # TODO: add spiflash1x support
-            spiflash_pads = self.platform.request("spiflash4x")
+            spiflash_pads = self.platform.request("spiflash")
+            #spiflash_pads = self.platform.request("spiflash4x")
             self.submodules.spiflash = SpiFlash(
                 spiflash_pads,
                 dummy        = dummy_cycles,

Generating the fbi files:

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/emulator$ python3 /home/pdp7/dev/enjoy/litex/build/lib/litex/soc/software/mkmscimg.py -f -o emulator.bin.fbi emulator.bin
pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/emulator$ cd ../buildroot/
pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/buildroot$ python3 /home/pdp7/dev/enjoy/litex/build/lib/litex/soc/software/mkmscimg.py -f -o rootfs.cpio.fbi rootfs.cpio
pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/buildroot$ python3 /home/pdp7/dev/enjoy/litex/build/lib/litex/soc/software/mkmscimg.py -f -o Image.fbi Image
pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv/buildroot$ python3 /home/pdp7/dev/enjoy/litex/build/lib/litex/soc/software/mkmscimg.py -f -o rv32.dtb.fbi rv32.dtb

[pdp7]

I added a prefix of 0x00010000 so flashing these files won't overwrite the gateware.

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv$ cat images.json
{
    "buildroot/Image.fbi":        "0xc0010000",
    "buildroot/rootfs.cpio.fbi":  "0xc0810000",
    "buildroot/rv32.dtb.fbi":     "0xc1010000",
    "emulator/emulator.bin.fbi":  "0x20010000"
}

I then try to flash the files, but LiteX BIOS prints junk.

pdp7@x1:~/dev/enjoy/linux-on-litex-vexriscv$ lxterm --images=images.json /dev/ttyUSB0 --speed=1e6 --no-crc --flash
[LXTERM] Starting....
�����lr��lf

           ��
���� __      �
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Feb 15 2020 22:36:54
 BIOS CRC passed (4cd8d44c)

 Migen git sha1: --------
 LiteX git sha1: --------

--=============== SoC ==================--
CPU:       VexRiscv @ 48MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  32768KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 76Mbps Reads: 96Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Flashing buildroot/Image.fbi to 0xc0010000 (5082540 bytes)...
[LXTERM] Upload complete (89.9KB/s).
[LXTERM] Flashing buildroot/rootfs.cpio.fbi to 0xc0810000 (4063240 bytes)...
[LXTERM] Upload complete (89.9KB/s).
[LXTERM] Flashing buildroot/rv32.dtb.fbi to 0xc1010000 (1932 bytes)...
[LXTERM] Upload complete (97.7KB/s).
[LXTERM] Flashing emulator/emulator.bin.fbi to 0x20010000 (9592 bytes)...
[LXTERM] Upload complete (88.9KB/s).
[LXTERM] Done.
KCCCCToo many consecutive errors, abortingLoading emulator.bin from flash...
Error: Invalid image length 0xffffffff
Booting from flash...
Error: Invalid image length 0xffffffff
No boot medium found

--============= Console ================--
litex> >g��#"���H�#$�#&���B����P#���c������%��cJ���%���%��%���%����%�����%�����%���#�%����$�%����#��%����#$#�%����#$���%����#$�#��%����#$�#���%����#$�#���%����#$�#�����%����#$�#����%����#$�#������%����#$�#����m�%����#$�#�����&�%����#$�#�����m�%����#$�#�����m&�%����#$�#�����m&�%����#$�#�����m&�%����#$�#�����m&�%����#$�#�����m&7�%����#$�#�����m&��%����#$�#�����m&7E�%����#$�#�����m&7���%����#$�#�����m&7�E�%����#$�#�����m&7�E���%����#$�#�����m&7�E�&�%����#$�#�����m&7�E���%����#$�#�����m&7�E��&�%����#$�#�����m&7�E��&�%����#$�#�����m&7�E��&��%����#$�#�����m&7�E��&�%����#$�#�����m&7�E��&�
Command not found
litex> ��]����ɷ'F���`#��#�����G#���#�t��
Command not found
litex> �#���7���<�@�        `�

@mithro @enjoy-digital any ideas?

[enjoy-digital]

@pdp7: the default addresses for flashing are: https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/bios/boot.c#L411-L414 (+ your SPI Flash offset). You can modify it, but it needs to be consistent between the BIOS and flashing script. At startup, the BIOS will then copy the data from SPI Flash to the SDRAM and Emulator RAM and jump to Emulator RAM. In your case, it seems you are flashing the Linux images (not using the default addresses) but also "flashing" the emulator.bin to the Emulator RAM which will not work. I need to update the Arty support with that, If you are not able to make it work, i'll try to update it tomorrow.

[pdp7]

I'm going to close this issue and address the effort to boot from flash in #42