RFC: Creating a synthesis or PnR benchmark

[TheProgrammerIncarnate]

Ian Cutress has expressed interest in adding something FPGA-related to his CPU benchmarking suite. Here's my idea: benchmark the PnR step of the design using a frozen version of NextPnR, a pre-synthesised design(s), and a constant seed. Would that be enough to ensure the execution workload is equal across different platforms? Also, any recommendations on which designs from yosys-bench to use? Dr Cutress said he would like a benchmark that takes 30 minutes or less on a quad-core machine. I'm planning on running a Boom core or a LiteX SoC design through to see how long it takes next time I get a chance to.

[mithro]

We have a https://github.com/SymbiFlow/fpga-tool-perf which is a tool for understanding the performance of various toolchains, where performance means multiple things;

• If a benchmark actually works
• The resulting highest fmax achievable for the design
• Resources taken (like CPU + memory) to produce the bitstream
• etc

The focus is on real world designs (for example LiteX SoCs, PicoSoC and OpenTitan) that people are using rather than synthetic benchmarks and wants to make sure that the output result could be usable.

We have a Google doc at https://j.mp/fpga-tool-perf-spec which includes various information about what we are targetting. Currently it is mostly targeted at the Xilinx 7 series as that is the flow we have the most options to compare around (We have 3 options for place around! Vivado, nextpnr and vpr!).

It would be nice to have support for ECP5, iCE40 and anything else in the future too but they are not a high priority.

GitHub

SymbiFlow/fpga-tool-perf

FPGA tool performance profiling. Contribute to SymbiFlow/fpga-tool-perf development by creating an account on GitHub.

Google Docs

SymbiFlow FPGA Tool Performance (Xilinx Performance)

SymbiFlow FPGA Tool Performance (for Xilinx Devices) This doc → https://j.mp/fpga-tool-perf-spec GitHub repo → https://github.com/SymbiFlow/fpga-tool-perf Modes Fixed fmax -- Looking at run time and memory usage Find fastest fmax -- Try and find the fastest possible Toolchains Synthe...

[mithro]

If you are more interested in benchmarks which aim to exercise a CPU (IE The fact the benchmark is an FPGA toolchain is mostly inconsequential) Verilog to Routing also has a pretty CPU intensive benchmarks. See https://docs.verilogtorouting.org/en/latest/tutorials/titan_benchmarks/ -- these range from O(10 minutes) to O(2 days) run time. See also https://docs.verilogtorouting.org/en/latest/dev/developing/#running-tests and https://docs.verilogtorouting.org/en/latest/dev/developing/#evaluating-quality-of-result-qor-changes

Running the Titan Benchmarks — Verilog-to-Routing 8.1.0-dev documentation

Commit Procedures — Verilog-to-Routing 8.1.0-dev documentation

Commit Procedures — Verilog-to-Routing 8.1.0-dev documentation

[TheProgrammerIncarnate]

Sweet! Thanks for the resources. Didn't know some of this stuff already existed.