gsomlo
commented
1 year ago The thing I find particularly confusing is that, whether I've built my bitstream with --cpu-mem-width 1 (LiteX provides AXIUpConverter) or with --cpu-mem-width 2 (Rocket variant has matching 128-bit MEM port width for e.g. nexys video), if I read some random sata or sdcard block (with sata_read X or sdcard_read Y) the data displayed is the same (for the same value of X or Y, but native rocket width matching is the only one that boots (passes "liftoff") if booted from sata or sdcard.
I therefore continue to have no idea how come netboot (no dma) works in both cases, but sdcard/sata booting only works for rocket variants that natively match LiteDRAM width, and not when LiteX provides its own AXIUpConverter.
n-kremeris
Dolu1990
enjoy-digital
First, let's start with a diagram of how RocketChip is wired into LiteX:
Rocket has a MEM port which we directly wire to LiteDRAM in a point-to-point AXI link.
MMIO (LiteX CSR) access is routed over a separate port wired to the LiteX wishbone bus.
Finally, DMA is routed through the RocketChip, which is supposed to maintain coherence between LiteDRAM and its internal caches.
RocketChip can have its various AXI port widths configured independently. We currently keep MMIO and DMA at a constant 64 bit width, and configure different (sub-)variants of MEM port width 64, 128, 256, and 512 bit width to match the various FPGA development boards' LiteDRAM port widths.
If the chosen Rocket (sub-)variant's MEM port doesn't match LiteDRAM's port width, Litex adds an AXIUpConverter and issues a warning: https://github.com/enjoy-digital/litex/blob/master/litex/soc/integration/soc.py#L1564
The problem: When using a rocket variant with a 64-bit-wide MEM port and a wider (128 or more) LiteDRAM (with the LiteX supplied AXIUpConverter), memory accesses by the rocket core itself seem to work fine, but DMA (booting/accessing LiteSDCard or LiteSATA) fails, i.e., the data loaded from external devices ends up corrupted. I have not (yet) figured out the nature of this corruption. So far, the symptom is that I can boot such a system via
netboot, but if I try loading the same binaries from sdcard or sata, the process hangs at "liftoff".The other problem is that when Rocket is augmented with its own internal L2 cache, it only works with a 64-bit wide MEM port, which precludes its use with SoCs using a wider LiteDRAM. Related tickets are here:
(The latter is me inquiring about assumptions/restrictions about Rocket's ports' relative widths).
While the rocket L2 cache's inability to work with wider MEM ports is a problem, so is the fact that the LiteX AXIUpConverter seems to break DMA transfers routed through the Rocket chip, which is what this ticket is about...
I'll try to do some more troubleshooting as I get spare cycles, but any help/ideas/suggestions are very much welcome!