Goals

ChiBench consists of a large collection of Verilog programs mined from open-source github repositories. The goal of this benchmark suite is to test and debug electronic design automation (EDA) tools, such as the Jasper Formal Verification Platform or Intel Quartus. To test your EDA tool, simply pass all the programs in the ChiBench collection to it, and see if it crashes. Below we list three examples of issues reported in this way:

• 2159: Verible's obfuscator crashes when reading a program that only contains the pragma directive.
• 2181: Verible's parser crashes instead of reporting syntax errors related to instantiation type.
• 2189: Verible's code formater crashes with syntactically valid input.

If you ever use ChiBench to find bugs in some EDA tool, we would appreciate it very much if you could reach out to us and report your experience. If you need help to set up the scripts to do this kind of exploration, feel free to reach out to us as well!

Technical Report

The process of constructing and curating this collection of benchmarks is described on this technical report. Cite it as:

@misc{Sumitani24,
  title={ChiBench: a Benchmark Suite for Testing Electronic Design Automation Tools}, 
  author={Rafael Sumitani and João Victor Amorim and Augusto Mafra and Mirlaine Crepalde and Fernando Magno Quintão Pereira},
  year={2024},
  eprint={2406.06550},
  archivePrefix={arXiv},
  primaryClass={cs.AR}
}

Provenance

ChiBench contains only programs that were originally distributed with some license. Thus, each program in the ChiBench suite contains, as a header comment, the original license of that specification, plus a link to the repository from where that code was obtained. Notice that these programs might use different licenses, given that they were extracted from different projects. On May 24th, 2024, the following licenses were used among the repositories mined to build the ChiBench collection:

Acknowledgment

This project is sponsored by Cadence Design Systems. Additionally, the different people involved in this project acknowledge the support of CNPq, FAPEMIG, and CAPES. Finally, thank UFMG's Department of Computer Science for making available the infrastructure necessary to carry out this project.