AWSteria: Create spec and implementation plan for adding L2 with coherence support for L1 and host-to-FPGA DMA

[rsnikhil]

Create a spec and implementation plan for coherent L2-based DMA access for AWSteria. Immediate goal is to support ability of host-based Virtio code to share DDR4 memory data structures (virtqueues, buffers) coherently with Flute. Coherent DMA will also be useful for broadening CHERI security story to I/O devices etc. (later).

Outline: Flute's L1 and the host-to-FPGA DMA will share an L2 cache with coherence support. We expect to borrow Toooba's L2 cache, which already has coherence support for multiple L1 clients and for DMA, and adapt it for Flute and AWSteria.

[rsnikhil]

Progress report on studying cache-coherent L2 cache in RISCY-OOO and planning how to extract it for our use in vanilla Flute and CHERI-Flute on AWS

• Realized that it needs a write-back L1, not write-through; fortunately we have that ready

• Grokked the details of the L2's interaction with environment.

  • Back-end: memory read/write of cache lines: we already have an AXI4 adapter for this from Toooba

  • Front-end: simple FIFO-like messages to L1

• Made plans for modifying our write-back L1:

  • Extend cacheline states from 'Invalid/Valid/Dirty' (= M, S, I) to MESI
  • Fix L1 cache-write logic to first request L2 to upgrade S->E
  • Fix L1 to serve downgrade M/E->S/I (with possible writeback) requests from L2
  • All these are quite easy to do

• Grokked the dependencies of L2 (following import chains which, transitively, bring in a lot of irrelevant stuff, including Connectal stuff, L1 stuff, TLB stuff, ...)

  Pruned it to just what's relevant for our use.

• In the pruned version, grokked the 'ifdef' variations (MIT RISCY-OOO has many, for side-channel attack mitigation, for strong and weak memory models, etc.) to understand what's relevant for us.

• Was reminded that it already has a 'coherent DMA' port. In RISCY-OOO this was used for program-loading via Connectal.

  In Toooba we adapted this to do program-loading via our Debug Module, for which we had already developed an AXI4 adapter. This should facilitate our connection to AWS DMA PCIs.

Status and next steps:

• Discuss with Cambridge how mods for tag support fit into all this

• In Vanilla Flute (@rsnikhil): ready to start implementation, per observations above

  • Mods to L1
  • L1-L2 interface connections
  • Redo system graph to separate out uncached/IO paths from cache paths
    • Connect L2 DMA port to AWS DMA PCIS

• In CHERI Flute (@gameboo):

  • Fix up L1 writeback cache to support tags
  • Track the remaining Vanilla Flute changes in CHERI Flute

[rsnikhil]

My previous message articulates the plan. Nothing to add after more reflection, and we have started executing the plan, so I'm closing out this issue. Progress will be reported in Issue #118.