Make another regtest that runs at least one of these designs on the 22 nm COFFE architecture you created, to ensure it still works and to check it is within QoR bounds. The vtr_reg_qor_chain regression should be a good template, and the procedure is documented at https://docs.verilogtorouting.org/en/latest/dev/developing/#adding-tests
#2: Adding testbench
if we make each benchmark a sub-directory I am pretty sure it will break the existing vtr_flow scripts that will run the regtests above. That could be fixed, but the faster way to check in the testbenches would be to make a /testbenches directory under the vtr_flow/benchmarks/verilog directory and put a README and testbenches (or subdirectories with testbenches) under it.
#3: Adding the design to CI/QoR regression
Seems like these should be added to the benchmarks/verilog directory and run as part of the vtr suite during QoR runs then.
#4: Managing ML benchmarks
I think we can make a benchmarks/ml directory and just put a README in it pointing to the designs in benchmarks/verilog and later titan/. That way we don't have to add a lot of new tests, and we can run these new designs alongside the existing Odin-II compatible designs (with our current scripts) and average QoR across the larger suite.
Basically this groups designs by the flow they go through, and then READMEs/lists define subsets and combinations of interest. I think that will be the more maintainable solution.
#5: Fixing/removing engineer, company, etc. in the banner of the design
Probably should either fill in as much of this as possible (engineer, company can be institution, etc.) or delete this.
vaughnbetz
commented
3 years ago
PR https://github.com/verilog-to-routing/vtr-verilog-to-routing/pull/1573 resulted in adding a TPU-like design to the VTR benchmark suite. But there are some action items that need to be done. Filing an issue to track these. @aman26kbm will work on these items.
Pasting actions from @vaughnbetz 's comments in that PR:
#1: Adding tests For tests, I suggest
#2: Adding testbench if we make each benchmark a sub-directory I am pretty sure it will break the existing vtr_flow scripts that will run the regtests above. That could be fixed, but the faster way to check in the testbenches would be to make a /testbenches directory under the vtr_flow/benchmarks/verilog directory and put a README and testbenches (or subdirectories with testbenches) under it.
#3: Adding the design to CI/QoR regression Seems like these should be added to the benchmarks/verilog directory and run as part of the vtr suite during QoR runs then.
#4: Managing ML benchmarks I think we can make a benchmarks/ml directory and just put a README in it pointing to the designs in benchmarks/verilog and later titan/. That way we don't have to add a lot of new tests, and we can run these new designs alongside the existing Odin-II compatible designs (with our current scripts) and average QoR across the larger suite. Basically this groups designs by the flow they go through, and then READMEs/lists define subsets and combinations of interest. I think that will be the more maintainable solution.
#5: Fixing/removing engineer, company, etc. in the banner of the design Probably should either fill in as much of this as possible (engineer, company can be institution, etc.) or delete this.