Toward a calyx testbench

[ethanuppal]

@nathanielnrn said that we don't have a way to drop in a harness and directly interface with calyx components to test them (please correct me if I'm misinterpreting this!). It seems quite useful to exercise this kind of lower-level control over the signals instead of just memory inspection in post. Please let me know if this infrastructure already exists and is flourishing.

In terms of execution, there could be a fud2 op --through tb tb.sources="tb1.py,tb2.rs". We could cocotb, which is quite standard, I believe, and whose AXI wrapper is already being used to test YXI stuff. There's also this rust tb inspired by cocotb, but it hasn't been updated for 2 years. (The issue with using cocotb directly is that boilerplate-y Makefiles have to be littered throughout the testing codebase.) And of course, we could use a verilator harness, but those are similarly clunky and also take forever to compile.

[ethanuppal]

It also might be worth seeing if we can integrate calyx somehow directly as a cocotb sim? Unless this doesn't make sense semantically. It seems that (following https://github.com/cocotb/cocotb/blob/master/src/cocotb_tools/makefiles/simulators/Makefile.verilator) we'd need to create a Makefile that runs the tool, which means we could just wrap icarus verilog and call it a day.

[rachitnigam]

We generally dump out the VCD for the design and query specific signals we care about. However, you are right: this is a pretty baroque approach to doing things and a more principled approach to doing this would be good.

I'm AFK so can't link but two things to look at:

• Spade has C++ bindings to make it easy to write harnesses for Spade generated designs
• Chisel/FIRRTL has the concept of "boring" (drilling sense) internal signals up to the top module for testing. We could do similar things potentially.

[ethanuppal]

Spade's harnesses seem like a thin wrapper around cocotb/verilator, as I suggested above. I also wrote a harness wrapper for my own language compiling to calyx. I also think it would be useful to test arbitrary components and not simply the top-level one.

[rachitnigam]

Do you have a specific design in mind for something like this?

[ethanuppal]

I think we can have a fud op that sends a calyx program through a bunch of test benches, whether they be cocotb benches or verilator harnessss (inferred maybe by file extension?). I'm not sure if this approach would allow you to test arbitrary components, but I have the feeling that verilator generates headers for all components.

[ethanuppal]

Talked with @jku20 about multi-input ops in fud2 and a potential current workaround of passing the extra data via the config.