RFC: re-using compiled testbenches

[ducky64]

This seems to be the most in-demand feature request...

Here, I'll sketch out several ideas with trade-offs, and we'll see if we can figure out how useful each one is. Note: these are API sketches for now, no guarantees these are implement-able.

Simple precompiled units

This one is pretty basic, provide an alternate API that separates the test(...) { c => ... } into two parts: elaboration / compilation, which returns an object, and the test body, which can be invoked on the object, multiple times, with different tests.

Main downside: you need a handle to the object to run tests on it, this could make it more complicated when you have several tests across different files that could share the same object. Possibly worked around with global variables, or a library around that mechanism.

Briefly discussed in #106

Mockup:

class BasicTest extends FlatSpec with ChiselScalatestTester {
  val precompiled = build(new StaticModule(42.U))  // same interface as module construction for test(...)

  it should "test a thing" in {
    precompiled.test { c =>
      c.out.expect(42.U)
    }
  }

  it should "test it again w/o recompiling" in {
    precompiled.test { c =>
      c.out.expect(42.U)
    }
  }
}

class AnotherTest extends FlatSpec with ChiselScalatestTester {
  val precompiled = build(new StaticModule(42.U))  // can't see inside BasicTest, so need to re-create it
  ...
}

Separation of interface (Chisel Circuit) and backend

A variation on the above, this further separates the build(...) into two parts: one part that would provide the Chisel elaborated Module as the interface, and another part that provides the backend.

Two potential uses are:

1. Manually load a backend from disk (eg, a precompiled Verilator binary, or pre-elaborated FIRRTL file for treadle), to avoid the recompilation time. The main difference from above is that the precompiled units can persist across JVM invocations. This would be an advanced user API, because it relies on the elaborated Module interface being in-sync with the loaded backend, and the loaded backend being up-to-date (if it's a Chisel design). All that being said, I'm not sure how often you want to run a different test suite where you haven't updated the RTL.
2. Testing non-Chisel RTL (by using a Chisel Module shim or BlackBox), possibly including post-syn netlists. One issue with using Module shims is that some features of ChiselTest rely on circuit data, eg checking combinational paths for inter-thread safety, or (in the hopefully-near-future) checking clock sources to get clocks on wires.

Possible solution to #34

Mockup:

val simulator = VerilatorBackend.loadFrom(...)

val precompiled = build(new StaticModule(42.U), simulator=simulator)  // relies on the programmer being correct in that the interface maps to the simulator, otherwise may either fail at runtime, or produce weird behavior
// or, since build(...) and test(...) should share an API
test(new StaticModule(42.U), simulator=simulator) { c=> ...}

Automatically managed precompiled units, on disk

An automatically managed version of the above. where the circuit is elaborated and hashed, and some database is checked for whether there's a cached precompiled unit. Might produce some savings for Verilator especially when RTL is not modified (how often is that?), but because it introduces persistent state that is not under explicit control, it could also be the source of unintuitive bugs.

However, it would be very difficult to avoid re-elaboration, since you would still need an elaborated Module as an API into the simulator, and you need a way to detect source changes without running the Scala generator. Neither is impossible (maybe the former can be serialized, and you might be able to use source modification times for the latter, maybe with some file dependency graph), but would seem nightmarish to get correct.

This potentially eliminates the need to separate the build and test APIs, since all the uses of build would be automatically handled by test. But lots of magic behind the scenes.

Anyways, I've heard this idea thrown around every once in a while, but I'm not sure if this is really a good idea. I'm not generally a fan of persistent state.

[anthonyabeo]

I, for one, will love to be able to create a single instance of a module and use it in several related test cases like so:

val mod = new MyModule()

test() {
    mod...
}

test() {
    mod...
}

such that the internal register state of MyModule will persist across all the test cases.