Add support for 64 bit DMA wishbone bus

[antonblanchard]

On Microwatt (a 64 bit CPU) we have to narrow/widen the 32 bit litesdcard DMA wishbone bus:

https://github.com/antonblanchard/microwatt/blob/84473eda1b54a03794b2f3d13748f42b52ead34e/soc.vhdl#L247

This adds extra complexity, but is also a performance issue. I hacked up litesdcard and litex to provide a 64 DMA bus, and integrated it into Microwatt. SD card read performance increased by 11%.

My changes to litex are at:

https://github.com/antonblanchard/litex/commit/7ffb53e81ba3c99a29591b461e3bcb4119ec0e32

And the changes to litesdcard are at:

https://github.com/antonblanchard/litesdcard/commit/12c15e30827a052fd71f889225d57ea3e14a717b

I haven't submitted this as a PR yet, because it spans two repos, doesn't handle other wishbone configurations and my migen knowledge is pretty terrible. Was hoping to get some feedback.

[enjoy-digital]

Thanks @antonblanchard. I assume you are using LiteSDCard through the generator and integrated in VHDL in the Microwatt project?

When used fully integrated with LiteX, the DMA interface indeed directly uses the data-width of the DMA interface of the CPU, data-width that has to be provided to the generator in your case since not known and that was indeed not supported.

With 64-bit CPUs and fully integrated LiteX flow, LiteSDCard was still using 32-bit Wishbone automatically narrowed/widened by adapters when integrated in the design. That's interesting to know the performance improvement when using native data-width, we could also propagate the data-width to LiteSDCard.

Your changes in LiteX/LiteSDCard look fine, maybe we can indeed discuss it more before doing a PR (or I could integrate your changes myself). I'm not able to look at it now but will be before the end of the month.

[enjoy-digital]

BTW, the bottleneck with the 32-bit datapath (and also probably 64-bit one) is probably due to the Wishbone interconnect nor supporting bursts. With burst support (which I think you are supporting in Microwatt) performance would probably be similar. There are plans to support Wishbone bursts in LiteX so this could be an occasion to add it.

[antonblanchard]

@enjoy-digital I am indeed using the LiteSDCard generator, but it would be great if both that and the fully integrated flow could eventually support a 64 bit DMA wishbone. Happy to talk later on in the month and make whatever changes needed.

Also great news wrt wishbone burst/pipelined mode, it will definitely help Microwatt, especially network performance. We currently have some logic to disable wishbone pipelining, but can easily remove that to test.

[enjoy-digital]

@antonblanchard: Sure, we'll try to improve both cases (generator and full integrated flow), I was just trying to understand the current flow. It's possible that using 32-bit Wishbone in LiteSDCard, adaptation to/from 64-bit in the generator and burst support will provide the best resources/performance compromise. Testing this on Microwatt will be a good way to evaluate and improve these aspects.

[mtdudek]

Some time ago I tried using 64-bit wishbone with LiteSDCard, and as far as I remember It worked, but there is an issue here https://github.com/enjoy-digital/litex/blob/78c1751c4781ffe156128748810ee4af85fee058/litex/soc/interconnect/wishbone.py#L211 as Interface() is created in 32bit mode, so this place require fix to use bus width to work.