Manifests as an alternative to Filelists

[seldridge]

With the recent addition of optionality to FIRRTL with layers and instance choices, a problem is brewing as it relates to knowing what files to include for a given compilation.

An example is helpful for the purposes of this discussion. Consider the following circuit which has one degree of optionality through layer A. The use of DontTouchAnnotations exist to ensure that modules and optionality are not removed:

FIRRTL version 4.0.0
circuit Foo: %[[
  {
    "class": "firrtl.transforms.DontTouchAnnotation",
    "target": "~Foo|Foo>c"
  },
  {
    "class": "firrtl.transforms.DontTouchAnnotation",
    "target": "~Foo|Bar>a"
  }
]]
  layer A, bind:

  module Bar:
    input a: UInt<1>
    output b: UInt<1>

    connect b, a

  public module Foo:
    input a: UInt<1>
    output b: UInt<2>

    inst bar of Bar
    connect bar.a, a
    connect b, bar.b

    layerblock A:
      node c = not(a)

Currently, there are three ways that such a circuit could be compiled:

1. Using single file Verilog output.
2. Using one-module-per-file Verilog output.
3. Different behavior occurs if the user has include DUT and Testbench directory annotations. However, these are not something that should be brought into the spec as this does not handle multiple public modules well. (This issue is intentionally trying to replace this behavior.)

Note that it is reasonable that there are other strategies that could be employed here that are not one of two extremes. I.e., private modules can be flexibly emitted anywhere and it is reasonable that they could be emitted in a way that minimizes the number of distinct files.

Using single-file output, we get the following. Comments with "FILE" indicate what file the following block of SystemVerilog code is placed:

// ----- 8< ----- FILE "Foo.sv" ----- 8< -----
module Bar(
  input  a,
  output b
);

  assign b = a;
endmodule

module Foo_A(
  input a
);

  wire c = ~a;
endmodule

module Foo(
  input        a,
  output [1:0] b
);

  wire _bar_b;
  Bar bar (
    .a (a),
    .b (_bar_b)
  );
  assign b = {1'h0, _bar_b};
endmodule

// ----- 8< ----- FILE "layers_Foo_A.sv" ----- 8< -----
// Generated by CIRCT firtool-1.66.0-95-g4996f6781
`ifndef layers_Foo_A
`define layers_Foo_A
bind Foo Foo_A a_0 (
  .a (a)
);
`endif // layers_Foo_A

When compiling with one-file-per-module output, you get the following files:

// ----- 8< ----- FILE "Bar.sv" ----- 8< -----
module Bar(
  input  a,
  output b
);

  assign b = a;
endmodule

// ----- 8< ----- FILE "Foo_A.sv" ----- 8< -----
module Foo_A(
  input a
);

  wire c = ~a;
endmodule

// ----- 8< ----- FILE "Foo.sv" ----- 8< -----
module Foo(
  input        a,
  output [1:0] b
);

  wire _bar_b;
  Bar bar (
    .a (a),
    .b (_bar_b)
  );
  assign b = {1'h0, _bar_b};
endmodule

// ----- 8< ----- FILE "layers_Foo_A.sv" ----- 8< -----

// Generated by CIRCT firtool-1.66.0-95-g4996f6781
`ifndef layers_Foo_A
`define layers_Foo_A
bind Foo Foo_A a_0 (
  .a (a)
);
`endif // layers_Foo_A

This creates problems for downstream consumers of this. Specifically:

1. If a user wants to compile module Foo, the ABI specifies that a file Foo.sv must exist. However, it does not say anything about where the other files are. The ABI is intentionally quiet about this for private modules and for layer enable files to allow for the compiler to place them anywhere it wants.
2. A user of a FIRRTL compiler may want to, after the compilation, reorganize the outputs to match the needs of a customer. Specifically, it is useful to organize the directory structure based on what is "test code" and what is "design code" or to put layer collateral in it's own area. The FIRRTL spec should not care about nor prescribe such a structure as this is expected to be different for different FIRRTL compiler users. Additionally, this requires the spec to answer questions related to the placement of a module instantiated by two directory areas. I would prefer if this was left to the user of a FIRRTL compiler and not something the FIRRTL spec has to prescribe.
3. Related to (2), it is often desirable for a FIRRTL compiler user to remove optionality after compilation. E.g., there is more verification code in some layers that is used to verify a piece of IP produced by a FIRRTL compiler that is removed when releasing this IP to a customer.
4. The ABI doesn't say anything about what to do for files required for optionality. E.g., when compiling Foo with layer A enabled, the ABI says how to enable the layer, but not that that will include all optional files. (Should it?)

A partial solution to these problems exists in the FIRRTL spec today through filelists on public modules. The FIRRTL spec states that each public module must provide a filelist for all modules it instantiates. It follows that such filelists must be emitted for every degree of optionality in a circuit---each layer or instance choice becomes another filelist. This is viewed as not desirable (due to the duplication it creates). Filelists are also not really lists of files. They are newline delimited command line arguments provided to tools (see verilator -F). (This means they are actually lists of positional arguments.) It is natural that these may want to expand to include actual arguments in the filelists. However, this will run into problems where while filelists are closer to being universally accepted by tools, the arguments to them are not.

As an alternative to this, and proposed by several others, it may be useful to add the notion of a FIRRTL manifest to the ABI. Such a manifest would provide information that would allow a user to answer the questions raised above and to, through lightweight tooling, create a directory structure that a user wants and generate command line arguments to drive downstream tools.

A manifest should include at least the following information:

1. The manifest version (as this may change over time as new FIRRTL spec or ABI features are added).
2. The list of all public modules in the circuit.
3. A description of all optionality in the circuit (layers, instance choices, and parameters once they are added).
4. An instance graph that describes the instance hierarchy in the design.
5. What file each module is in.
6. Whether or not an instance is optionally instantiated and with what optionality feature.

As an example of this, for the single-file compilation, such a manifest could be:

{
    "version": "0.1",
    "name": "Foo",
    "instanceChoices": [],
    "layers": [
        {
            "name": "A",
            "convention": "bind",
            "file": "layers_Foo_A.sv",
            "layers": []
        }
    ],
    "parameters": [],
    "publicModules": [
        {
            "name": "Foo",
            "convention": "v1"
        }
    ],
    "instanceGraph": [
        {
            "module": "Foo",
            "file": "Foo.sv",
            "instances": [
                {
                    "module": "Bar",
                    "instance": "bar",
                    "conditional": null
                },
                {
                    "module": "Bar",
                    "instance": "bar",
                    "conditional": {
                        "type": "layer",
                        "layer": "A"
                    }
                }
            ]
        },
        {
            "module": "Foo_A",
            "file": "Foo.sv",
            "instances": []
        },
        {
            "module": "Bar",
            "file": "Foo.sv",
            "instances": []
        }
    ]
}

For one-file-per-module emission, the manifest would be:

{
    "version": "0.1",
    "name": "Foo",
    "instanceChoices": [],
    "layers": [
        {
            "name": "A",
            "convention": "bind",
            "file": "layers_Foo_A.sv",
            "layers": []
        }
    ],
    "parameters": [],
    "publicModules": [
        {
            "name": "Foo",
            "convention": "v1"
        }
    ],
    "instanceGraph": [
        {
            "module": "Foo",
            "file": "Foo.sv",
            "instances": [
                {
                    "module": "Bar",
                    "instance": "bar",
                    "conditional": null
                },
                {
                    "module": "Bar",
                    "instance": "bar",
                    "conditional": {
                        "type": "layer",
                        "layer": "A"
                    }
                }
            ]
        },
        {
            "module": "Foo_A",
            "file": "Foo_A.sv",
            "instances": []
        },
        {
            "module": "Bar",
            "file": "Bar.sv",
            "instances": []
        }
    ]
}

With one of these manifests, aforementioned lightweight tooling can generate a filelist based on arguments to the tool. E.g., a hypothetical tool invocation could be manifest-tool filelist manifest_Foo.json --top-module Foo --layer A and return one of the following filelists for the different manifests above:

Foo.sv
layers_Foo_A.sv

Foo.sv
Bar.sv
Foo_A.sv
layers_Foo_A.sv

Related Efforts

This is very similar to existing non-standard files emitted by the SFC and the MFC. Cf. dut_hier.json and tb_hier.json which are JSON representations of the instance hierarchy under, respectively, the design-under-test and the testbench.

This is also likely similar to existing industry standard formats like IP-XACT or similar efforts like DuH.

[darthscsi]

Filelists are also not really lists of files. They are newline delimited command line arguments provided to tools (see verilator -F).

Is this true for other tools? It looks like it is true for VCS.

[darthscsi]

I'm not that thrilled about exposing an instance graph per-se.

[darthscsi]

With layers, though not instance choice, the file list can be inclusive (no name conflicts arise from including more files than you need).

[darthscsi]

I also don't think single-file emission is something we care about. I was inclined to hide it.

[seldridge]

I discussed this with @darthscsi offline last week. The conclusion was that it would be better to not go the route of filelists or manifests. Instead, we should move in the direction of a flat output structure. Any movement of files into directories (which is still needed) can be handled by FIRRTL compilers in a post-pass. If this post-pass needs information that would be contained in a filelist/manifest, then it is the responsibility of the FIRRTL compiler to provide access to this. For CIRCT, this can be via an API that queries the MLIR instance graph. Other compilers may choose to implement a filelist or manifest if they choose.