[dv] Ibex co-simulation proposal

GregAC commented 3 years ago

Our current Ibex DV environment uses a post simulation trace comparison methodology. This checks that the traces match from a particular test binary that is run on both an ISS (such as Spike or OVPSim) and an Ibex simulation. It has a weakness around interrupts, exceptions and debug related behaviour. These aren't simulated on the ISS (with the exception of some precise exceptions such as an illegal instruction, we don't simulate precise exceptions from imem or dmem error responsess). This will cause trace mismatches so trace matching is disabled on tests that use these with final state comparison and some extra checking during the simulation run to compensate. This can suffice for finding bugs but it can require more tests to ensure a particular bug actually produces a failure. When failures are seen debugging back from a final state comparison failure can be hard work.

This is a proposal to switch to a co-simulation method where Spike is run along with the Ibex DV simulation in lock step, cross checking between the two on every retired instruction. The DV environment will probe events such as interrupts, debug requests and memory error responses. These will be notified to Spike to keep instruction execution in sync.

Note this proposal specifically concerns using Spike as the ISS for co-simulation. Currently OVPSim is also supported for the post simulation trace comparison method but due to the higher degree of coupling we should focus on supporting a single ISS for co-simulation. The co-simulation framework should be designed such that support for multiple simulators in future is possible without significant refactoring.

I have done some initial investigation around how to directly control the Spike 'core' and inserting interrupts/memory errors/debug requests at appropriate points so the instruction by instruction trace matches an Ibex simulation. This has demonstrated that co-simulation with Spike is feasible.

Spike Modifications

The initial investigation has shown minimal Spike source modifications are required. Two changes have been identified:

Alter the debug entry point which is specified by the DEBUG_ROM_ENTRY define in debug_rom_defines.h. We may wish to alter this to a run-time parameter or at least provide compile-time configurability (given it needs to match an Ibex parameter DmHaltAddr which can be altered in the Ibex instantiation).
Make register writes visible to the public interface of processor_t. When Spike is built with RISCV_ENABLE_COMMITLOG register writes are already logged however this happens internally to the step function and the publically visible state that would contain register writes is wiped (having been output to the textual log) before step returns. An alternative would be to ignore this functionality all together and simply look at the state of the register file after every step and check for changes.

Spike uses a autoconf/automake based build system. For my initial investigation I used meson to build specific Spike source files as only a subset were required. For the co-simulation framework we should seek to use as much of the existing Spike build infrastructure as possible. There are two possibilities:

Use the existing Spike build unaltered, it produces a libriscv.a that I believe contains everything that will be required for the co-simulator. However it may also bring up some other things we don't need. Further investigation is needed to see if this can be used as is.
Add a new target to the Spike build system (e.g. libspike.a or libcospike.a) that provides precisely what we need for the co-simulator.

It should be noted Spike's way of specifying instruction requires some build time generation of C files from template files. This is a simple process using sed driven by the autoconf/automake generated Makefiles. It could be recreated but it is preferable to use the original Spike build system to do the generation.

Ideally any modifications we do need to make will be upstreamed so we should choose what we do with that in mind.

Spike Usage

Spike is written in C++ using a number of classes to implement its functionality. The co-simulator will be based around the processor_t class which represents a single processor and provides functionality for step by step instruction execution. The Spike README states:

The C++ interface to Spike's internals is not considered a public API at this time, and backwards-incompatible changes to this interface will be made without incrementing the major version number.

This is unfortunate as ideally we'd use something that would be considered stable as the base for our co-simulation framework. Instead we should aim to interact with Spike internals in the most minimal way possible ideally using the public interfaces of classes where possible (e.g. avoid adding new methods to expose previously private aspects of classes).

processor_t exposes a state_t via a public get_state function which gives read/write access to the processor state including the register file, PC and MIP (interrupt pending CSR). It also contains a log of register writes and memory accesses however this isn't of much use without modification as it gets internally written, relevant log entries made, then wiped, before it can be observed from a public interface.

processor_t has a public step function which will be used at the core of the co-simulation framework. This steps the simulation by one or more instructions. It will only be used to step by a single instruction with appropriate checks before and after to ensure the step matches the corresponding Ibex instruction retirement.

To avoid altering how the commit log functionality works (required to get commit log information outside of step) we could implement our own step. The function is relatively simple and this would allow us to access and clear the commit log as required however this deals more with the private aspects of the class implementation so may be more likely to see breaking changes.

An implementation of the simif_t interface must be provided to each processor_t instance, this provides the simulated process with access to memory. Internally spike uses a TLB like setup where the first time it wants to access a byte within a 4k page is asks a simif_t instance for a pointer to it (using the addr_to_mem method). Following accesses have the pointer cached in the TLB and do not call into the simif_t interface. This is good for simulation performance but not useful for co-simulation, in particular where we need to simulate memory errors on specific addresses at specific times.

This TLB mechanism can be bypassed by returning NULL from addr_to_mem, this causes Spike to use the mmio_load/mmio_store functions to perform the memory accesses directly. For co-simulation we will always return NULL from addr_to_mem so each individual memory access (both data and instruction) will result in a call to mmio_load or mmio_store. Precise exceptions from memory errors can be stimulated by returning false from mmio_load or mmio_store.

The co-simulator will keep its own copies of the simulated memories which will be preloaded with the same contents as the memories from the Ibex simulation but otherwise will remain independent.

processor_t header: https://github.com/riscv/riscv-isa-sim/blob/ce170a71eb1ab5296b547fdeafe40c9448e9cdda/riscv/processor.h#L267 simif_t header: https://github.com/riscv/riscv-isa-sim/blob/master/riscv/simif.h

Co-simulation DPI library

Spike will be implemented inside a library exporting C functions to be used over DPI from the RTL simulator. Functions provided will be:

riscv_cosim_init - Initialises Spike and provides a chandle referring to the simulator instance that needs to be used with all other functions.
riscv_cosim_destroy - Deallocates and does any require clean up for the simulator instance chandle
riscv_cosim_step - Steps the simulator one instruction and compares results against provided instruction results from Ibex RTL instruction retirement. The function returns 1 if the step succeeds and matches and 0 otherwise. Things to check (passed as arguments to riscv_cosim_step) are:
- PC of executed instruction
- Register written and value written
riscv_cosim_set_mip - Sets the mip CSR (interrupts pending), Spike will immediately take an interrupt at the next riscv_cosim_step if a pending interrupt is enabled.
riscv_cosim_set_dbg_req - Sets/clears the Spike debug halt request. Spike will immediately enter debug mode at the next riscv_cosim_step if the debug halt request is set.
riscv_cosim_notify_mem_err - Takes a 32-bit aligned address as an argument. The next time Spike accesses that any byte within the 4-byte region a memory error should be signalled (by returning false from mmio_load/mmio_store).

Functions will also be required for memory setup and preloading, these are TBD (as there's various options but ultimately straight-forward so wish to keep discussion focussed on the co-simulation aspects).

Do we want some functions to help compare CSR changes? It's possible to get several in one step (e.g. when taking a trap) so it can't be a simple CSR written with value passed to to riscv_cosim_step. We could provide separate riscv_cosim_notify_csr_write that does the checking/batches up notifications to check at the next step?

How do we want to communicate details of ISS/RTL mismatches? The co-sim library could just directly write to standard out or a specified log file but that's not great. Perhaps a separate riscv_cosim_get_error which returns a mismatch error string?

Ibex Integration

The existing RVFI interface can provide instruction PCs and register file writes suitable for passing to riscv_cosim_step. Some extra tracing infrastructure will be required to deal with interrupts and debug requests. Currently the Ibex micro-architecture won't jump into an interrupt or debug handler until the instruction present in ID/EX when the interrupt/debug request arrives has moved from ID/EX into WB (or retired in the 2 stage design). In effect the state of MIP or debug request the last cycle the instruction is in IF is the relevant one. We could extend rvfi to provide this. The following signals would be added:

rvfi_ext_mip - State of MIP the cycle before the instruction entered ID/EX (the MIP that should apply to that instruction, in particular if it's the first time an enabled interrupt is seen we would expect the instruction to be the first instruction of the interrupt handler). rvfi_ext_dbg_req - As rvfi_ext_mip but for debug request.

The DV environment would call riscv_cosim_set_mip and riscv_cosim_set_dbg_req with these values before calling riscv_cosim_step. We could wrap in the mip and debug request status into the riscv_cosim_step arguments but I think it's preferable to retain some flexibility here should there be any scenarios this scheme doesn't work for.

DV environment integration

As a C DPI library we should able to use the co-simulation framework from many simulators, in particular Verilator and VCS. Plain systemverilog can be used to drive the co-simulation which will work equally well in a SimpleSystem Verilator setup as well as the existing UVM environment run in VCS or other commercial simulations (the approach used by the OTBN co-simulation setup).

I think it may be worth doing a UVM version, using a UVM monitor to probe the RVFI with a seperate UVM component taking the monitor results and driving the co-simulator DPI interface. As we develop the co-simulator framework we may encounter scenarios where we need some detailed interactions between the co-simulator and UVM testbench which will be better served if the co-simulator is part of the UVM testbench rather than a seperate Verilog component. Alternatively we could proceed with the plain systemverilog version only and switch to a UVM version should we encounter a need for it.

Memory Access Checking

We should add a checker that probes the Ibex DMem interface and ensures all transactions match expected memory accesses (this is more complex on some processors as there will be speculative memory accesses, currently all DMem accesses on Ibex relate to instructions that will retire). The spike internal commit logging already has some logging of memory reads and writes which could be used to cross-check. Using calls to the mmio_load/mmio_store functions to cross check against Ibex accesses is tricky as you can't determine if a particular mmio_load is for an instruction fetch or a DMem read.

GregAC commented 2 years ago

This has now been implemented.

Jiahua-Gong commented 1 year ago

Hi,I met a error. Can you help me ?


ibex/dv/cosim/spike_cosim.cc", line 66: error: class
          "processor_t" has no member "enable_log_commits"
      processor->enable_log_commits();

spike_cosim.cc", line 231: error: class "state_t"
          has no member "last_inst_pc"
    if (processor->get_state()->last_inst_pc == PC_INVALID) {
                                ^
spike_cosim.cc", line 332: error: class "state_t"
          has no member "log_reg_write"
    auto &reg_changes = processor->get_state()->log_reg_write;
                                                ^

xmsc: Error executing: $CDSROOT/tools/cdsgcc/gcc/6.3/bin/g++ -DNCSC -DCADENCE  -DLNX86 -D_GLIBCXX_USE_CXX11_ABI=0 -I$CDSROOT/tools/systemc/include_pch/64bit -o $TESTDIR/out/build/tb/xcelium.d/run.lnx8664.20.09.d/xmsc_run/xmsc_obj/spike_cosim_0.o -I/test_jiahua/ibex/dv/cosim -I/test_jiahua/test2/newspike/include -I/test_jiahua/test2/newspike/include/softfloat -I$CDSROOT/tools/include -I$CDSROOT/tools/inca/include -DXMSC -DNCSC -I$CDSROOT/tools/systemc/include_pch -I$CDSROOT/tools/tbsc/include -I$CDSROOT/tools/vic/include -I$CDSROOT/tools/methodology/OVM/CDNS-2.1.2/sc/src -I$CDSROOT/tools/methodology/UVM/CDNS-1.2/sc/sc -I$CDSROOT/tools/methodology/UVM/CDNS-1.2/ml/sc -I$CDSROOT/tools/systemc/include/cci -I$CDSROOT/tools/systemc/include/factory -I$CDSROOT/tools/systemc/include/tlm2 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0 -c -x c++  -Wall /test_jiahua/ibex/dv/cosim/spike_cosim.cc
xmsc_run: *E,TBBLDF: Failed to generate object /test_jiahua/ibex/dv/uvm/core_ibex/out/build/tb/xcelium.d/run.lnx8664.20.09.d/xmsc_run/xmsc_obj/spike_cosim_0.o

xrun: *E,CCERR: Error during cc compilation (status 1), exiting.

lowRISC / ibex