Test configurations

[ducky64]

This arose out of support for -DwriteVcd=1 in ScalaTest being broken, and whether that breakage was intentional or not. At the chisel-dev meeting, this turned into a longer discussion about configurations, in particular addressing #65.

There seems to be agreement about using annotations as the internal mechanism for representing configurations, including (as currently supported) dumping VCDs and simulator backend.

The overall design requirements are:

• Should apply for multiple tests (or all tests implicitly?)
• Parameters are visible prior to test execution (?)
• Repeatable and deterministic
• Usable interface
• Familiar interface (ideally reusing existing infrastructure)

That being said, there was a debate on what the interface should look like.

Proposal 1: universal configurations in a separate file

Under this proposal, there would be a configuration file, probably in the resources folder, formatted as annotations command line syntax, that would be read in to provide default annotations for all tests.

Pros:

• No test modification needed to change existing tests. If you wanted all tests to behave a certain way without changing each test individually, this would do it.
• Single underlying mechanism. I guess aside from the per-test .withAnnotations(...).
• Independent of test infrasructure (eg, ScalaTest), since it's handled at the ChiselTest infrastructure level

Cons:

• The configuration file isn't obvious, unless you read the docs (which I wouldn't assume). Kind of mutually exclusive with no test changes needed to add the global configuration, since otherwise you would need to add some kind of reference to the additional annotations in each test. But you don't get the reference that tells you that a non-default configuration is being loaded.
• Additional state separate from your .scala files.
• No separation of temporary (eg, waveform dumping) and permanent (eg, use Verilator) configurations.
• Only one global configuration.

I personally think the cons outweigh the pros here, especially since the configuration is somewhat implicit.

Proposal 2: two mechanisms for two purposes

This proposal provides both a temporary mechanism (akin to the debug printf - something that isn't pushed to version control) for changing global configurations, and a permanent mechanism (something that does get pushed to version control) for sharing annotations across multiple test classes while remaining in Scala.

Temporary (per-run) annotations through test infrastructure arguments

Basically, bring back (and possibly expand) the -DwriteVcd=1 option that can be passed to ScalaTest in sbt.

Pros:

• Extremely lightweight. No mucking around creating files in your resources folder.
• Temporary configurations are structurally temporary. Your source files don't get changed, there's nothing to revert before you commit, and you can't accidentally commit the equivalent of a debug printf.

Cons:

• Uncertain how this works for non-sbt/ScalaTest environments - but you can always fall back to .withAnnotations(...).
  • In particular, it's unclear how this works with IntelliJ/ScalaTest. @chick apparently didn't have success configuring a default test configuration for ScalaTest that works across multiple unit tests. But I'd be kind of surprised if IntelliJ actually had no way of setting a project-wide default test configuration.

Permanent annotations in the class hierarchy

Currently, tests are largely written with

class BasicTest extends FlatSpec with ChiselScalatestTester { ...

This proposal would allow subclassing ChiselScalatestTester with annotations, for example:

class MyTestConfig extends ChiselScalatestTester(annos=Seq(writeVcdAnnotataion))
...
class BasicTest extends FlatSpec with MyTestConfig { ...

Potential solution for #65.

Pros:

• The existence of eg MyTestConfig in the unittest class hierarchy provides a dead giveaway of some non-default configuration being applied, and IDEs can furthermore navigate to that link.
• Allows multiple configurations per project, with all the features of inheritance.
• Pure Scala, consistent with existing withAnnotations(...), managed by existing infrastructure like sbt.

Cons:

• It's a bit harder to add a global config to all tests, since you have to update the inheritance hierarchy of each test. Though it's trivial to update the shared config class afterwards.

Misc

Still up for debate how to run multiple backends. Should this be a responsibility of the build system (and above the ChiselTest level)? Or should there be an annotation that runs a single test multiple times under different backends? Should all tests be written such that they can run multiple times in the same JVM invocation? How would this coexist with proposals to avoid multiple elaboration?

[ducky64]

@seldridge @chick @ucbjrl @azidar @jackkoenig since you were all part of that discussion

[chick]

I think this is a good summary. I am open to any approach that brings this capability to the stack. I'd like to add that there be a diagnostic interface, .i.e. a way to get the relevant Stage to report what options are in play and where they are coming from.

[ekiwi]

I see test configuration management as out of scope for chiseltest. Users should be able to create their own solutions on top of the testing APIs that we provide.