search

enjoy-digital / litedram

Small footprint and configurable DRAM core
Other
372 stars 120 forks source link

Add ECP5 support to standalone core generator #106

Closed ximinity closed 4 years ago

ximinity commented 4 years ago

I would like to use the LiteDRAM core as a standalone core within my designs. I'm using the Versa ECP5-5G board as my development platform. This repository contains a gen.py file that exists to perform exactly these steps. However, only the Xilinx-7 series of FPGAs are currently supported. I would like to add ECP5 support to this generator.

I've looked at the versa_ecp5 repository to see how to instantiate the core. I've tried to add the LiteDRAM configuration in that repository to the gen.py generator. The verilog that is generated synthesizes using yosys/nextpnr/prjtrellis and can be flashed to the FPGA using openocd. However, the init_done and init_error are never asserted.

105 is a WIP implementation for the ECP5 generator. I've added an extra configuration in the examples directory for the ECP5. It can be invoked in the same way as the other configurations:

gen.py ../examples/versa_ecp5.yml.

Aside from the generator changes I would like to ask why the init_done and init_error signals are never asserted and what actions need to be performed for this work (in the case of a standalone core).

enjoy-digital commented 4 years ago

Thanks for the initial implementation! Adding ECP5 support to the generator was planned and i will help getting your it working.

enjoy-digital commented 4 years ago

@ximinity as discussed in https://github.com/enjoy-digital/litedram/pull/105 your initial PR has been merged and modified a bit to make it more generic, thanks for your work!

While doing it, the only real issue i saw is that you disabled the CPU, but the CPU is needed for the calibration. In the future, we will try to use the smallest CPU possible for that to avoid wasting resources. Could you do another test with upstream code?

ximinity commented 4 years ago

@enjoy-digital Thanks for the changes and the merge!

I noticed that you changed the default toolchain target for ECP5 to diamond for Lattice's Diamond toolchain. I use yosys/nextpnr/prjtrellis as my toolchain. If I change the toolchain in the new gen.py it fails at the following step:

$ ./gen.py ../examples/versa_ecp5.yml
...
python3 -m litex.soc.software.mkmscimg bios.bin --little
make: Leaving directory '/home/ximin/litex/litedram/litedram/build/software/bios'
Traceback (most recent call last):
  File "./gen.py", line 596, in <module>
    main()
  File "./gen.py", line 575, in main
    vns      = builder.build(build_name="litedram_core", regular_comb=False)
  File "/usr/lib/python3.8/site-packages/litex/soc/integration/builder.py", line 181, in build
    vns = self.soc.build(build_dir=os.path.join(self.output_dir, "gateware"),
  File "/usr/lib/python3.8/site-packages/litex/soc/integration/soc_core.py", line 512, in build
    return self.platform.build(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/litex/build/lattice/platform.py", line 34, in build
    return self.toolchain.build(self, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/litex/build/lattice/trellis.py", line 216, in build
    (family, size, speed_grade, package) = nextpnr_ecp5_parse_device(platform.device)
  File "/usr/lib/python3.8/site-packages/litex/build/lattice/trellis.py", line 80, in nextpnr_ecp5_parse_device
    size        = device.split("-")[1]
IndexError: list index out of range

If I specifically use the litex.boards.platforms.versa_ecp5's Platform class then the generation works. I noticed that the nextpnr_ecp5_parse_device function that throws the exception was modified in enjoy-digital/litex@83ad674feb21098c0e51e2d1c70dbdffb1744cd5, perhaps it is related.

enjoy-digital commented 4 years ago

@ximinity: the generated code will be the same, i just changed the toolchain as a workaround for the issue you just have. I'm planning to improve that, but you can already test the generated code in your design.

rowanG077 commented 4 years ago

@enjoy-digital Where can I find the code that does the sdram calibration? I would like to implement this in hardware so a CPU is not strictly needed.

enjoy-digital commented 4 years ago

@rowanG077: the SDRAM initialization C headers are generated by https://github.com/enjoy-digital/litedram/blob/master/litedram/init.py and the initialization happens in https://github.com/enjoy-digital/litex/blob/master/litex/soc/software/bios/sdram.c#L963. Initializing up a SDR/DDR/DDR2 SDRAM should not be too complicated to do in pure logic, but for DDR3/DDR4 doing the read/write leveling is more complicated and using a CPU is easier does not necessarily cost more resources than the additional logic that will be needed to do it in pure logic. The best for that is to use the smallest CPU possible. Lite variant of VexRiscv is already good for that, in the future we'll probably have smaller variants or cores that we could use and switch to it will be very easy. So unless you are not going to use DDR3 and DDR4, i'm not sure it's worth doing it it in pure logic.

rowanG077 commented 4 years ago

@enjoy-digital Is it crucial read/write leveling happens every boot? I'm kind of starved for resources on my platform so if I make a bootloader with a simple CPU that does the leveling then store the values in flash and read them out would that also be fine?

enjoy-digital commented 4 years ago

@ximinity: if the board is operated in the same conditions, it should indeed work. The calibration can also be done over a UART to Wishbone bridge when the board is connected to a Host. (https://github.com/enjoy-digital/versa_ecp5/blob/master/test/test_sdram.py) but that won't work if your FPGA is working in standalone.

ximinity commented 4 years ago

@enjoy-digital

As you've said before, the soft-CPU inside the LiteDRAM core needs perform the initialization. The litedram_core with the CPU option set exposes the serial receive and transmit pins. The serial rx and tx are then constrained to the FPGA Pins as specified in the Versa documentation. I'm new to LiteX so I'm not exactly sure what to do here in order to make initialization work. Currently I do the following:

  1. Create my own design.
  2. Instantiate litedram_core in it.
  3. Synthesize it using yosys/nextpnr/trellis.

In other examples, such as [1], you also need to build firmware and upload it using, for example, litex_term. Do I need to do the same here, e.g. create 'firmware' for the CPU that:

  1. Defines a main function.
  2. Calls the SDRAM initialization functions through the headers that are generated by the standalone core generator.
  3. Compiles it similar to the firmware in [1].
  4. Loads it similar to load_firmware.py in [1].

[1] https://github.com/enjoy-digital/versa_ecp5

enjoy-digital commented 4 years ago

Sorry @ximinity for the delay. The standalone generator also generates the initial rom content that will calibrate the memory. When building with Yosys/Nextpnr, just make sure to provide litedram_core.v and litedram_core.init (which is the ROM contents). When loading your design, you should see the LiteX bios prompt and DDR3 calibration on the UART.

enjoy-digital commented 4 years ago

@ximinity: have you been able to do another test? If not, can you share your Versa ECP5 project so that i can do some hardware tests? @rowanG077: If you are operating your system in similar conditions, your could indeed use pre-computated levelings values.

ximinity commented 4 years ago

@enjoy-digital

Sorry for the late response. I was originally planning on doing the testing for both this and the LiteEth changes at the start of this week but I've been really busy with writing a report for an assignment. The report is due tomorrow so I'm planning on doing the testing on Saturday. I'll report the test results and project samples then as well. Thanks for all the help you've been providing!

ximinity commented 4 years ago

@enjoy-digital

I've tested the new changes and attached the basic example that I used. It simply wires the init_done and init_error to the LEDs on the Versa board. I've used the latest versions of LiteX and LiteDRAM. The only change I've made to the versa_ecp5.yml is to use just the wishbone interface. After uploading the design to the board I get the follow output: image

The example files I tested with: sdram_example.tar.gz.

It contains the generated core and some additional files. The wrapper instantiates a PLL and the LiteDRAM core and connects the result.

enjoy-digital commented 4 years ago

@ximinity: thanks for the test and files. Can you do another test with https://github.com/enjoy-digital/litedram/commit/b8339886dabc54cc40b69d081ea544c6c88b682a? The calibration was not setup correctly for ECP5 and this is probably what is causing the issue.

ximinity commented 4 years ago

@enjoy-digital

I've tested the changes but the result is the same: image

There are less attempts (it stops after the second try) but the amount of errors are the same.

enjoy-digital commented 4 years ago

OK thanks. Not sure i tested at this frequency on ECP5, could you try to increase to 75MHz? If it's still not working, i'll do some tests on hardware.

ximinity commented 4 years ago

@enjoy-digital

Same result for 75MHz: image

enjoy-digital commented 4 years ago

OK, i'll do some test on hardware. Can you also provide the .yml file you are using? It seems the MAIN_RAM size is not the expected one. (IIRC it should be 128MB with the Versa ECP5).

ximinity commented 4 years ago

I've used the examples/versa_ecp5.yml as a base, with some small modifications (only wishbone and 75MHz as the system clock). The input_clk_freq doesn't seem to be used for the ECP5.

{
    # General ------------------------------------------------------------------
    "cpu":        "vexriscv",  # Type of CPU used for init/calib (vexriscv, lm32)
    "memtype":    "DDR3",      # DRAM type

    # PHY ----------------------------------------------------------------------
    "sdram_module":    "MT41K64M16",  # SDRAM modules of the board or SO-DIMM
    "sdram_module_nb": 2,             # Number of byte groups
    "sdram_rank_nb":   1,             # Number of ranks
    "sdram_phy":       "ECP5DDRPHY",  # Type of FPGA PHY

    # Frequency ----------------------------------------------------------------
    "input_clk_freq":   100e6, # Input clock frequency
    "sys_clk_freq":     75e6,  # System clock frequency (DDR_clk = 4 x sys_clk)
    "init_clk_freq":    25e6,  # Init clock frequency

    # Core ---------------------------------------------------------------------
    "cmd_buffer_depth": 16,    # Depth of the command buffer

    # User Ports ---------------------------------------------------------------
    "user_ports": {
#        "axi_0" : {
#            "type": "axi",
#            "id_width": 32,
#        },
        "wishbone_0" : {
            "type": "wishbone",
        }
#        "native_0" : {
#            "type": "native",
#        },
#        "fifo_0" : {
#            "type":  "fifo",
#            "base":  0x00000000,
#            "depth": 0x01000000,
#        },
    },

    # CSR Port -----------------------------------------------------------------
    "csr_expose": "False", # Expose CSR bus as I/Os
    "csr_align" : 32,      # CSR alignment
}
enjoy-digital commented 4 years ago

@ximinity: sorry for the delay, i wanted to look at it but i'm having trouble compiling the example design you provided with up to date Yosys/Trellis/Nextpnr:

ERROR: timing analysis failed due to presence of combinatorial loops, incomplete specification of timing ports, etc.

Would you mind checking it compiles with up to date tools and eventually update it if not? Thanks.

ximinity commented 4 years ago

@enjoy-digital

I was able to replicate the issue with new versions of yosys/nextpnr/trellis. I've updated my toolchain to the following commits:

It seems there was an issue with the flags that I passed to yosys. I've added the updated example: sdram_example.tar.gz. The only thing you should need to modify is the sdram_example/wrapper.ys file. The read_verilog calls contain some absolute paths for my system.

enjoy-digital commented 4 years ago

Ok thanks. I'll look at that on next monday.

enjoy-digital commented 4 years ago

I did some changes to litedram_gen (https://github.com/enjoy-digital/litedram/commit/d4d9ab740e873a4b14b8a3a015445fbc8652fbf2, https://github.com/enjoy-digital/litedram/commit/f6babda6831277976018f734610a24d1cf648f46) and with the attached project it's working but not reliably: some initialization are failing, some are working while we don't have this behavior with the LiteX targets. I'll stop for today on this, but will have another look in the next days. Could you do some tests on your side to see if you have the same behaviour?

sdram_example_2020_03_26.zip

enjoy-digital commented 4 years ago

Here is a log of both cases:

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Mar 26 2020 17:18:24
 BIOS CRC passed (7fafd584)

 Migen git sha1: 3f9809b
 LiteX git sha1: 4abb3715

--=============== SoC ==================--
CPU:       VexRiscv @ 50MHz
ROM:       24KB
SRAM:      4KB
L2:        0KB
MAIN-RAM:  16384KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under software control
Read leveling:
m0, b0: |11100000| delays: 01+-01
best: m0, b0 delays: 01+-01
m1, b0: |11100000| delays: 01+-01
best: m1, b0 delays: 01+-01
SDRAM now under hardware control
Memtest bus failed: 2/256 errors
Memtest data failed: 524282/524288 errors
Memtest addr failed: 32/8192 errors
Memory initialization failed

--============= Console ================--
litex> 
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2020 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Mar 26 2020 17:18:24
 BIOS CRC passed (7fafd584)

 Migen git sha1: 3f9809b
 LiteX git sha1: 4abb3715

--=============== SoC ==================--
CPU:       VexRiscv @ 50MHz
ROM:       24KB
SRAM:      4KB
L2:        0KB
MAIN-RAM:  16384KB

--========== Initialization ============--
Initializing SDRAM...
SDRAM now under software control
Read leveling:
m0, b0: |11100000| delays: 01+-01
best: m0, b0 delays: 01+-01
m1, b0: |11100000| delays: 01+-01
best: m1, b0 delays: 01+-01
SDRAM now under hardware control
Memtest OK
Memspeed Writes: 202Mbps Reads: 141Mbps

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
No boot medium found

--============= Console ================--
litex>
ximinity commented 4 years ago

@enjoy-digital I can confirm the same behavior, though a successful memtest is very rare. The memtest bus/data/addr section now also give different results each run.

enjoy-digital commented 4 years ago

@ximinity: thanks for confirming this. We don't have this behaviour when built directly with LiteX, i'll try to understand.

enjoy-digital commented 4 years ago

With the recent changes and upstream repositories, the issue is no longer occurs. This has been tested with previous sdram_example_2020_03_26.zip and regenerated controller.