[RFC] Create Backend Plug-In API

[chick]

This is a proposal for a new API that allows us and others to create new backends for simulations and other purposes. It is in driven by to RFC #725, which moves chisel-testers functionality into chisel3.

Motivation

Chisel-testers currently supports 3 backends

• verilator
• VCD
• interpreter For release 3.2 #725 will move most of the chisel-testers functionality into chisel3 but currrently does not include support for any of the chisel-tester supported backends, it instead uses treadle. It is likely that we will encounter more desired backends in the future.

There is far to much duplicated boiler-plate in the existing backend support code and the move to chisel3 from chisel-testers is an ideal time to implement this.

Goal

A single API that wraps the backend functionality of all the existing backends in a coherent structure and provides and interface for us and others to add new backends with a minimum of pain.

Proposal

Backend Plug-In API

This will define an API for

• run-time creation of a backend
• hooks for initializing and parameterizing the backend
  • which should tie into our state of the art execution options
• executing the backend
• programming interface for creating new backends.

Initial Scope

Initial list of supported backends

• FIRRTL emission
• Verilog emission
• verilator simulation
• VCS simulation
• Interpreter simulation
• Treadle simulation'
• Treadle REPL
• GUI simulation

Extensibility

New backends can be added using a library based model that can be added through the standard SBT library dependency interface.

Parameterizable

Probably the trickiest part of this is that most backends will have specific custom parameterization requirements. The VCS and verilator backends, for instance, require the ability to pass down nearly arbitrary command flags to the underlying compilers. Treadle and the Interpreter, have large run-time options that require many flags with some overlap and some possible name collisions. This project should inform and benefit from the Options as Annotations work

[chick]

@seldridge @jackkoenig I've been meaning to take a look at this since summer began. As I started writing it up, I am not sure, but I think that this may have requirements that could affect some of the decisions begin made with the annotations as options implementation. Possibly more complications for the lazy val problem.

[seldridge]

Related narrowly to the problem of conflicting flags... this came up with Verilator/VCS and riscv-fesvr. As there was no good approach for mangling there, the solution that we developed was the use of +permissive/+permissive-off to ignore option parsing within specific regions. But, yes, there's a general problem here of the top-level option parser cannot be expected to know what the backend options are (nor would the implementer of a new Bakend Plug-in have any idea about) and these backends like to use very weird, non-POSIX options that are just not well behaved. At an annotation level, it would seem reasonable that each plug-in would define an annotation for storing options that should be directed to that plug-in.

Yes, definite heavy annotations/options overlap. I'll think about this more, but it would seem great if there's some feedback on the annotations/options refactor based on some implementation of an exotic backend (or maybe walking through a sketch of what this would look like). I'm strongly in favor of getting the annotations/options approach right definitively and more use cases can only help that.

Specific items that come to mind:

• Multiple backends/emitters/plug-ins should be capable of being simultaneously targeted with the same run (the simple solution which I don't believe works now is adding multiple emitter annotations, e.g., for high, mid, low, and verilog output)
• This may benefit from @azidar's non-RFC'd ideas about automatic transform (or backend/emitter/plug-in) scheduling

[seldridge]

One approach here would be to build out a type hierarchy off of Transform and to use AnnotationSeq as the universal means of communication. This would redefine Transform to not be CircuitState => CircuitState, but AnnotationSeq => AnnotationSeq. In effect, generalizing Transform. There are three four main children that extends Transform:

• Frontends serve as source nodes
• Transforms are FIRRTL-internal transforms
• CustomTransforms are intermediate nodes supplied by external libraries
• Backends are sink nodes

A frontend includes additional capabilities to read directly from the command line. This would enable every frontend to be assembled into a fat jar to work as a standalone executable.

/** A node in a transform graph that exists in FIRRTL-proper */
private[firrtl] abstract class Transform {
  def execute(annos: AnnotationSeq): AnnotationSeq
}

/** A custom FIRRTL transform. These are what external libraries provide. */
abstract class CustomTransform extends Transform

/** A frontend. This is a source node in a transform graph. 
  * 
  * This a firrtl.options.Driver is now in tha options as annotations refactor. 
  */
abstract class Frontend extends Transform {
  val optionsManager: ExecutionOptionsManager
  def execute(args: Array[String], annos: AnnotationSeq): AnnotationSeq
  def main(args: Array[String]): AnnotationSeq
}

/** A backend. This is a sink node in a transform graph. */
abstract class Backend extends Transform

Within this type hierarchy, Chisel's Driver is a Frontend, a FIRRTL-internal transform is a Transform, an external transform (e.g., FAME-1) is a CustomTransform, an emitter is a Backend. Nodes that are multiple things are somewhat ambiguous. Chisel's current Driver is currently a frontend, a transform, and a backend (as it calls FIRRTL explicitly and runs it). This may motivate breaking things out to do less, i.e., make driver's narrowly act as frontends.

[seldridge]

One concrete SO post talking with a use case:

• https://stackoverflow.com/questions/51933441/passing-arguments-to-verilator-backend-in-chisel

[seldridge]

I think that any solution here should also be thinking about backends of backends. The natural example here would be HAMMER. When ASIC and FPGA toolflows enter the picture, the Frontend/Compiler/Backend notion becomes less clear as these run after the Verilog backend. Nonetheless, abstracting all "tools" inside a firrtl.options.Driver or similar would provide an elegant abstraction.

[jackkoenig]

This is done with Stage/Phase / chiseltest.