sim: use SDRAM DFI model for simulation

enjoy-digital commented 4 years ago

Requires https://github.com/enjoy-digital/litex/issues/352, https://github.com/enjoy-digital/litex/issues/353, https://github.com/enjoy-digital/litex/issues/355, https://github.com/enjoy-digital/litedram/issues/102 and https://github.com/enjoy-digital/litex/issues/356.

Currently, the simulation use a very large RAM inferred from verilog as the main Memory. With the new features added to litex_sim we should be able to boot linux in simulation with LiteDRAM + SDRAM LPDDR/DDR/DDR2/DDR3 or DDR4 model. This simulation will be more representative of the behaviour on hardware.

enjoy-digital commented 4 years ago

Added with https://github.com/litex-hub/linux-on-litex-vexriscv/commit/14059868095915995846e5dc0269ea2b69f31261 and booting :), congrats @piotr-binkowski for the DFI model work!:

       __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jan 31 2020 13:34:40
 BIOS CRC passed (5a0dccd8)

 Migen git sha1: d11565a
 LiteX git sha1: 7a6c04db

--=============== SoC ==================--
CPU:       VexRiscv @ 1MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  65536KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 0Mbps Reads: 0Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x20000000

--============= Liftoff! ===============--
VexRiscv Machine Mode software built Jan 31 2020 13:34:33
--========== Booting Linux =============--
[    0.000000] No DTB passed to the kernel
[    0.000000] Linux version 5.0.13 (florent@lab) (gcc version 8.3.0 (Buildroot 2019.08-rc2-00011-gad9efda578)) #1 Thu Sep 12 14:20:26 CEST 2019
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Initial ramdisk at: 0x(ptrval) (8388608 bytes)
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x00000000c0000000-0x00000000c3ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000c3ffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000c0000000-0x00000000c3ffffff]
[    0.000000] elf_hwcap is 0x1101
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16256
[    0.000000] Kernel command line: mem=64M@0xc0000000 rootwait console=liteuart earlycon=sbi root=/dev/ram0 init=/sbin/init swiotlb=32
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Sorting __ex_table...
[    0.000000] Memory: 52300K/65536K available (3415K kernel code, 148K rwdata, 509K rodata, 140K init, 216K bss, 13236K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.000405] sched_clock: 64 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
[    0.004720] Console: colour dummy device 80x25
[    0.007332] Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
[    0.008986] pid_max: default: 32768 minimum: 301
[    0.026111] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.027632] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.113045] devtmpfs: initialized
[    0.169535] random: get_random_bytes called from setup_net+0x4c/0x188 with crng_init=0
[    0.180102] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.182407] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.198871] NET: Registered protocol family 16
[    0.526150] clocksource: Switched to clocksource riscv_clocksource
[    1.013132] NET: Registered protocol family 2
[    1.042660] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes)
[    1.044615] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    1.047090] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    1.048521] TCP: Hash tables configured (established 1024 bind 1024)
[    1.054979] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    1.056569] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    1.086159] Unpacking initramfs...
[    5.175665] workingset: timestamp_bits=30 max_order=14 bucket_order=0
[    6.029263] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    6.031152] io scheduler mq-deadline registered
[    6.032038] io scheduler kyber registered
[    9.059943] f0001000.serial: ttyLXU0 at MMIO 0xf0001000 (irq = 0, base_baud = 0) is a liteuart
[    9.062451] printk: console [liteuart0] enabled
[    9.062451] printk: console [liteuart0] enabled
[    9.063401] printk: bootconsole [sbi0] disabled
[    9.063401] printk: bootconsole [sbi0] disabled
[    9.112449] libphy: Fixed MDIO Bus: probed
[    9.123513] i2c /dev entries driver
[    9.233369] NET: Registered protocol family 10
[    9.274194] Segment Routing with IPv6
[    9.278799] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    9.368380] Freeing unused kernel memory: 140K
[    9.368951] This architecture does not have kernel memory protection.
[    9.370347] Run /init as init process
mount: mounting tmpfs on /dev/shm failed: Invalid argument
mount: mounting tmpfs on /tmp failed: Invalid argument
mount: mounting tmpfs on /run failed: Invalid argument
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Initializing random number generator... [   14.884344] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: OK
Starting dropbear sshd: [   17.658352] random: dropbear: uninitialized urandom read (32 bytes read)
OK

Welcome to Buildroot
buildroot login: root
                __   _
               / /  (_)__  __ ____ __
              / /__/ / _ \/ // /\ \ /
             /____/_/_//_/\_,_//_\_\
                   / _ \/ _ \
   __   _ __      _\___/_//_/ __             _
  / /  (_) /____ | |/_/__| | / /____ __ ____(_)__ _____  __
 / /__/ / __/ -_)>  </___/ |/ / -_) \ // __/ (_-</ __/ |/ /
/____/_/\__/\__/_/|_|    |___/\__/_\_\/_/ /_/___/\__/|___/

 32-bit VexRiscv CPU with MMU integrated in a LiteX SoC

login[81]: root login on 'console'

root@buildroot:~#

enjoy-digital commented 4 years ago

@piotr-binkowski: it seems that with https://github.com/enjoy-digital/litedram/commit/99227ad0d0f5682d9e9a4f71ac306162c64e87f1 and https://github.com/litex-hub/linux-on-litex-vexriscv/commit/e6ef794c1bcf7b9615fe9efd40742d8e28292235, the contents is not initialized correctly since CPU detects an illegal instruction and terminate the simulation:

./sim.py --with-sdram

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Feb  3 2020 13:15:20
 BIOS CRC passed (17de72b9)

 Migen git sha1: 4d4d055
 LiteX git sha1: bd6fd3da

--=============== SoC ==================--
CPU:       VexRiscv @ 1MHz
ROM:       32KB
SRAM:      4KB
L2:        8KB
MAIN-RAM:  65536KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 0Mbps Reads: 0Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x20000000

--============= Liftoff! ===============--
VexRiscv Machine Mode software built Feb  3 2020 13:15:18
--========== Booting Linux =============--
- dut.v:6429: Verilog $finish

piotr-binkowski commented 4 years ago

Thanks for testing this, I did tests with ./sim.py --with-sdram --sdram-module MT41K128M16 so I didn't notice that the basic configuration has some issues. I will look into this right now.

enjoy-digital commented 4 years ago

Can you eventually verify that linux start booting with?: ./sim.py --with-sdram --sdram-module AS4C16M16 --sdram-data-width=16 (SDR Minispartan6) ./sim.py --with-sdram --sdram-module MT47H64M16 --sdram-data-width=16 (DDR2 Nexys4DDR) ./sim.py --with-sdram --sdram-module MT41K128M16 --sdram-data-width=16 (DDR3 Arty) ./sim.py --with-sdram --sdram-module K4B2G1646F --sdram-data-width=32 (DDR3 NeTV2y) ./sim.py --with-sdram --sdram-module EDY4016A --sdram-data-width=64 (DDR4 KCU105) (No need to do the full boot, just seeing the first lines of the linux prompt is enough).

piotr-binkowski commented 4 years ago

I did those tests that you requested and here are the results:

Minispartan6 uses 16-bit SDR memory which is not supported at the moment (PHY model needs support for converting 32-bit init into 8/16-bit sized chunks)
Nexys4DDR started booting fine
Arty started booting fine
NeTV2 started booting fine

KCU105 boots fine with --sdram-data-width=16 but with 64 fails with:

[pbinkowski@arch linux-on-litex-vexriscv]$ ./sim.py --with-sdram --sdram-module EDY4016A --sdram-data-width=64
Traceback (most recent call last):
File "./sim.py", line 224, in <module>
main()
File "./sim.py", line 200, in main
soc = SoCLinux(i!=0,
File "./sim.py", line 137, in __init__
self.register_sdram(
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_sdram.py", line 103, in register_sdram
self.register_mem("main_ram", self.mem_map["main_ram"], wb_sdram, main_ram_size)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_core.py", line 471, in register_mem
self.add_wb_slave(address, interface, size)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_core.py", line 363, in add_wb_slave
address_decoder = mem_decoder(address_or_address_decoder, size)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/common.py", line 18, in mem_decoder
assert (address & (size - 1)) == 0
AssertionError

and with 32 there is a memory region conflict:

[pbinkowski@arch linux-on-litex-vexriscv]$ ./sim.py --with-sdram --sdram-module EDY4016A --sdram-data-width=32
Traceback (most recent call last):
File "./sim.py", line 224, in <module>
main()
File "./sim.py", line 200, in main
soc = SoCLinux(i!=0,
File "./sim.py", line 137, in __init__
self.register_sdram(
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_sdram.py", line 103, in register_sdram
self.register_mem("main_ram", self.mem_map["main_ram"], wb_sdram, main_ram_size)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_core.py", line 472, in register_mem
self.add_memory_region(name, address, size)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_core.py", line 468, in add_memory_region
self.add_mem_region(name, length, type)
File "/home/pbinkowski/projects/litedram/litex/litex/soc/integration/soc_core.py", line 457, in add_mem_region
raise ValueError("Memory region conflict between {} ({}) and {} ({})".format(
ValueError: Memory region conflict between csr (<SoCMemRegion 0xf0000000 0x10000 cached>) and main_ram (<SoCMemRegion 0xc0000000 0x40000000 cached>)

Should I look into adding support for 8/16-bit SDR SDRAM?

enjoy-digital commented 4 years ago

Ok thanks. This is fine for the KCU105, the fails are not related to the DFI model, not need to look further for it, i'll do that. For 8/16-bit SDR SDRAM, that would be nice yes. This way we would be able to simulate all configurations (The Hackaday badge with the SDRAM extension module uses a 8-bit SDR SDRAM for example).

piotr-binkowski commented 4 years ago

With enjoy-digital/litedram#129 it is possible to run Minispartan6 config and even ./sim.py --with-sdram --sdram-data-width=8 --sdram-module AS4C32M8 that uses 8-bit SDR memory

enjoy-digital commented 4 years ago

Great, thanks. We can now use the DFI model with all our hardware configurations so we can close this.

litex-hub / linux-on-litex-vexriscv

sim: use SDRAM DFI model for simulation #84