search

enjoy-digital / litex

Build your hardware, easily!
Other
2.99k stars 568 forks source link

Understand BRAM usage #1112

Closed francis2tm closed 2 years ago

francis2tm commented 2 years ago

Hello, I'm using the platform orangecrab and I've built a SoC using litex. I'm taking advantage of orangecrab's external SDRAM and thus I'm not using integrated main ram (integrated_main_ram_size = 0). The utilization report done by trellis indicates (printed to terminal during build, btw is there a log where this info can also be found?):

Info: Device utilisation:
Info:          TRELLIS_SLICE:  7444/12144    61%
Info:             TRELLIS_IO:    82/  197    41%
Info:                   DCCA:     4/   56     7%
Info:                 DP16KD:    46/   56    82%
Info:             MULT18X18D:     4/   28    14%
Info:                 ALU54B:     0/   14     0%
Info:                EHXPLLL:     2/    2   100%
Info:                EXTREFB:     0/    1     0%
Info:                   DCUA:     0/    1     0%
Info:              PCSCLKDIV:     0/    2     0%
Info:                IOLOGIC:    49/  128    38%
Info:               SIOLOGIC:     0/   69     0%
Info:                    GSR:     0/    1     0%
Info:                  JTAGG:     0/    1     0%
Info:                   OSCG:     1/    1   100%
Info:                  SEDGA:     0/    1     0%
Info:                    DTR:     0/    1     0%
Info:                USRMCLK:     1/    1   100%
Info:                CLKDIVF:     1/    4    25%
Info:              ECLKSYNCB:     1/   10    10%
Info:                DLLDELD:     0/    8     0%
Info:                 DDRDLL:     1/    4    25%
Info:                DQSBUFM:     2/    8    25%
Info:        TRELLIS_ECLKBUF:     3/    8    37%
Info:           ECLKBRIDGECS:     1/    2    50%

We can conclude that Litex is using 46 BRAMs.. Which seems a lot. I looked around in the source and the only components that I think can contribute to the BRAM count are: ROM, SRAM and L2 cache. Although these only take 131072 8192 and 8192 bytes respectively, which would represent 9 BRAMs in total, if I'm not mistaken. Where can I get more utilization info to determine what's causing such high BRAM usage?

Also, SRAM and cache have suspiciously the same size, is SRAM = cache?

Thanks in advance, and great work with litex

enjoy-digital commented 2 years ago

Hi @francis2tm,

During the build, the ROM is automatically reduced to the minimal size but you can maybe play with the --integrated-sram-size and --l2-size command line parameters to see if changes have the expected impact on BRAM usage.

Can you provide the command you are using for the build, I could also do some tests.

enjoy-digital commented 2 years ago

Closing since initial question has been answered and no feedback.

francis2tm commented 2 years ago

Hello @enjoy-digital, Sorry for only returning now to this issue. By decreasing the l2-size to 0, I get BRAM utilization of 27/56. I think the ROM occupies 14 BRAMs, the other 13BRAMs I don't know where they're being used. I'm using the https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/gsd_orangecrab.py target.

So now I have 2 questions:

  1. Is it possible to move the BIOS, that currently is in ROM, to the rest of the firmware that resides in the SPI FLASH?
  2. Besides the l2 cache and ROM, what is using up those 13 BRAMs? Perhaps valentyusb core?
francis2tm commented 2 years ago

Hello @enjoy-digital, Sorry for only returning now to this issue. By decreasing the l2-size to 0, I get BRAM utilization of 27/56. I think the ROM occupies 14 BRAMs, the other 13BRAMs I don't know where they're being used. I'm using the https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/gsd_orangecrab.py target.

So now I have 2 questions:

1. Is it possible to move the BIOS, that currently is in ROM, to the rest of the firmware that resides in the SPI FLASH?

2. Besides the l2 cache and ROM, what is using up those 13 BRAMs? Perhaps valentyusb core?

@enjoy-digital this is the new question I left here before you reopened the issue

francis2tm commented 2 years ago

Any thoughts regarding this? @enjoy-digital

cklarhorst commented 2 years ago

Hey @francis2tm

you said in your first post:

I looked around in the source and the only components that I think can contribute to the BRAM count are: ROM, SRAM and L2 cache. Although these only take 131072 8192 and 8192 bytes respectively

I don't use the orangecrab, so I might be wrong here, but: 131072 bytes for the bios looks very much like Auto-Resizing doesn't work! At least for the digilent_basys3.py target, the default ROM size is 0x20000 => 131072 bytes. Which is then resized to something below 30 KB.

For example, my log (digilent_basys3.py) shows: INFO:SoC:Auto-Resizing ROM rom from 0x20000 to 0x5ca4

Besides the l2 cache and ROM, what is using up those 13 BRAMs? Perhaps valentyusb core?

The selected CPU might come with L1 I/D caches. For example, the default VexRiscV CPU has 4096B I and 4096B D-Cache.

francis2tm commented 2 years ago

Hey @cklarhorst, Yes, I believe that's the problem. When I run the target script: python3 gsd_orangecrab.py I don't see any message related to the Auto-Resizing... Isn't it enabled by default? If not, how do I enable the Auto-Resize of the bios? The output related to rom:

INFO:SoCBusHandler:rom Region added at Origin: 0x00000000, Size: 0x00020000, Mode: R, Cached: True Linker: False.
INFO:SoCBusHandler:rom added as Bus Slave.
INFO:SoC:RAM rom added Origin: 0x00000000, Size: 0x00020000, Mode: R, Cached: True Linker: False.
cklarhorst commented 2 years ago

I just run the gsd_orangecrab target on the current master branch. I found out that you need to use --build so that it compiles the software & hardware, otherwise the message will not be printed.

The last lines before it starts the hardware toolchain should look like:

[-> after picolibc build]
 CC       crt0.o
 CC       bios.elf
chmod -x bios.elf
 OBJCOPY  bios.bin
chmod -x bios.bin
python3 -m litex.soc.software.crcfbigen bios.bin --little
python3 -m litex.soc.software.memusage bios.elf /homes/cklarhor/tmp2/build/gsd_orangecrab/software/bios/../include/generated/regions.ld riscv64-unknown-linux-gnu

ROM usage: 26.56KiB     (20.75%)
RAM usage: 1.61KiB      (20.12%)

rm crt0.o
make: Leaving directory '/homes/cklarhor/tmp2/build/gsd_orangecrab/software/bios'
INFO:SoC:Initializing ROM rom with contents (Size: 0x6a50).
INFO:SoC:Auto-Resizing ROM rom from 0x20000 to 0x6a50.

=> The LiteX-SoC overview only shows the default size => Resizing happens after SW build just before the HW is build

I also checked the resulting design (build/gsd_orangecrab/gateware/gsd_orangecrab.v) and it looks correct (6804*4 = 27216 B):

//------------------------------------------------------------------------------
// Memory rom: 6804-words x 32-bit
//------------------------------------------------------------------------------
// Port 0 | Read: Sync  | Write: ---- |
reg [31:0] rom[0:6803];
initial begin
        $readmemh("gsd_orangecrab_rom.init", rom);
end
francis2tm commented 2 years ago

Yeah, now I also see it! Thanks a lot. Yes my BIOS, even after auto-resizing, uses up 14/56 BRAMs... Since BIOS is only ran at startup time, is there any way of moving the BIOS to a flash memory? Or at least a way to reuse the ROM after BIOS has been ran?

cklarhorst commented 2 years ago

Hi @francis2tm

1409 should save you a few more BRAMs.

Since BIOS is only ran at startup time, is there any way of moving the BIOS to a flash memory?

I'm not sure but I don't think that this is currently supported.

Or at least a way to reuse the ROM after BIOS has been ran?

I think you can already put the BIOS into SRAM instead of ROM (in the end, both are implemented via the FPGAs block ram). But you would lose the ability to reboot, it is unsafe because you should not override the ISR region and I have no idea what to store there.

enjoy-digital commented 2 years ago

Hi,

just some precisions:

It's possible to run the BIOS directly from SPI Flash, as done for example on some boards like the iCEBreaker, it requires:

It's also possible to switch the ROM to a pre-initializated SRAM by setting integrated_rom_mode to "rw":

francis2tm commented 2 years ago

Hi,

just some precisions:

It's possible to run the BIOS directly from SPI Flash, as done for example on some boards like the iCEBreaker, it requires:

* Disabling the integrated ROM:  https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/icebreaker.py#L80

* Adding the SPI Flash support to the SoC: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/icebreaker.py#L98-L101

* Defining the ROM region (in SPI Flash) and set CPU reset to it: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/icebreaker.py#L103-L109

It's also possible to switch the ROM to a pre-initializated SRAM by setting integrated_rom_mode to "rw":

* https://github.com/enjoy-digital/litex/blob/master/litex/soc/integration/soc_core.py#L81

Thanks, this is exactly what I needed!