How to run the Koios benchmarks for QoR

[vaughnbetz]

Expected Behaviour

We should have a clearly documented procedure for running the Koios benchmarks for QoR.

Current Behaviour

It appears the benchmarks are split over vtr_reg_nightly6, vtr_reg_weekly and vtr_reg_nightly4. The documentation at https://docs.verilogtorouting.org/en/latest/README.developers/#example-koios-benchmarks-qor-measurement just says to run vtr_reg_nightly6 and vtr_reg_weekly.

Possible Solution

If those are the only benchmarks we care about, we should say so and say why. If we care about all of them, we should have vtr_reg_nightly4 in the list too.

Another minor thing to fix: the comment in the code snippet at https://docs.verilogtorouting.org/en/latest/README.developers/#example-koios-benchmarks-qor-measurement says "run the VTR benchmarks". It should say "Run the Koios benchmarks"

Context

Generating results for a paper and making sure we have the full Koios benchmark suite run in the proper way.

Your Environment

VTR master as of Sept. 25.

[aman26kbm]

We do want all of them. Will fix the QoR documentation.

Looks like the documentation at https://github.com/verilog-to-routing/vtr-verilog-to-routing/tree/master/vtr_flow/benchmarks/verilog/koios also needs to be fixed.

[saaramahmoudi]

@aman26kbm As I have made a mistake while running koios (I was running part of it since I didn't know about all tasks), can we at least mention in the documentations why did we split it? If you agree, I can modify all documents (including Vaughn's suggestion).

[aman26kbm]

The reason for the split was that the older Koios benchmarks could run with the default ODIN-only synthesis flow, but most of the newer Koios 2.0 benchmarks needed the Yosys+ODIN flow. So, we created new regression suites/tests for the ones that needed the Yosys+ODIN flow (vtr_reg_nightly6 and in vtr_reg_weekly). They differed in the cmake arg "ODIN_USE_YOSYS" (in .github/workflows/test.yml).

https://github.com/verilog-to-routing/vtr-verilog-to-routing/blob/37502f5b6ad0c6882455135374693291c540d79b/.github/workflows/test.yml https://github.com/verilog-to-routing/vtr-verilog-to-routing/pull/2165#issuecomment-1264239715

I agree this should have been clarified properly in the documentation.

I see that things have evolved since then. There is a new default flow called parmys. I don't see the ODIN_USE_YOSYS thing in test.yml anymore. I see that we now have separate regressions with "_odin" in their name (with a new cmake arg added to those i.e. WITH_ODIN=ON). These are the ones that worked with ODIN only.

I don't see weekly tests in test.yml though. Please check the settings for those.

Thanks for offering to fix the documentation, @saaramahmoudi. Appreciate it.

[vaughnbetz]

We should probably test with parmys (the default) now.