CI setup for RTL benchmarks

[tangxifan]

Problem Definition

Currently, the Github action is only set up for notifying the gap analysis repo: https://github.com/RapidSilicon/Gap-Analysis However, for RTL benchmarks, CI is need to certify the RTL benchmarks by ensuring

• They can be successfully compiled by different tools, including
  • Modelsim (for verification tools)
  • iVerilog (for verification tools)
  • Yosys
  • Vivado synthesis
  • Quartus synthesis
  • Anlogic synthesis
  • Gowin synthesis

If any benchmarks cannot be compiled as is, CI should report error and we can assign engineers to fix the issue

• If any RTL benchmark has self-checking testbenches, we should run the HDL simulation to ensure that benchmarks work as expected.

If any benchmarks cannot pass HDL simulation, CI should report error and we can assign engineers to fix the issue

• Any pull request must pass these tests before merged into the master branch

Possible Solution

Github runner set-up:

• We can use the Github runners for iVerilog and Yosys compilation test. On CI runner, we should display the version of iVerilog and Yosys. See the implementation in OpenFPGA: https://github.com/lnis-uofu/OpenFPGA/blob/30feb78469dc6132c0d66485ba4e91254b0de1a6/.github/workflows/build.yml#L95
• We have to use a custom runner (any machine in Rapidsilicon) for tests using Modelsim, Vivado, Quartus etc. I believe @zafarali-rs has already experiences in setting up the CI in this way.

How many Github Runners are needed

• Remember that the github runner provided by Microsoft has a limited hardware resource (up to 7GB RAM, 14GB diskspace and 6 hour runtime, https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources)
• We want to speed up the CI as much as we can. Therefore, We need to split the workload on Github runners evenly. I suggest to control the CI runtime within 30 minutes. It means all the jobs will finish in 30 minutes. If we see any exception, we should create a new runner.

[komaljaved-rs]

We are working on it.

[komaljaved-rs]

CI setup is up on the master branch for the 4 tools i.e. Yosys Vivado Quartus Lattice Diamond

All the tools run independently on the self hosted runners and check if the added/modified benchmark is synthesizable or not.

[tangxifan]

@komaljaved-rs Thanks for the updates. I just take a quick look and there are several issues.

• The CI is running when there are pushes on two branches, master and cga_test. Later, should we constrain only the master branch? Since we configured the CI to be run on any pull requests
• I did not see a pull request on the master branch. I remember that you did add protection rules on the master branch (as discussed in #2 ) . Does it mean that the protection rules are not set up correctly?
• Did you set any github checks required when merging a pull request?

• For all the tests, I see 1 benchmark ac97 is used. That makes sense to me. However, the log files are very long. I see you have python scripts to run the tests. Can you redirect the tool output to log files? In the CI log, we just need to see if a benchmark is o.k. or not. For example,

[laraibkhan-rs]

@tangxifan rules were set correctly before. I have set them again.

[komaljaved-rs]

• Yes, I'll remove the cga_test branch from the CI tests once whole flow will be setup. cga_test branch is kept for testing purpose.

• OK I'll turn of the logs display avoid lengthy logs.

[tangxifan]

@laraibkhan-rs I think you are going in a right direction. However, I cannot see any pull request when you push changes to the master branch. Does it mean there are still someone who can directly modify the master branch without going through code review stages?

[laraibkhan-rs]

@tangxifan it had been deactivated as we were doing some small changes in designs renaming and in scripting. I have set rule again and a pull request in created. You can check it out.

[tangxifan]

@laraibkhan-rs Understood. This makes sense to me. Since we now have CI, we should accept pull requests as the only way to modify the master branch now.

I think this issue is close to be addressed. The remaining work is on

• The python script to run each tools and generate reports
• Ensure all the benchmarks are covered in CI

After the CI setup, we should start developing

• Issue template. Refer to https://github.com/RapidSilicon/OpenFPGA_RS/tree/master/.github/ISSUE_TEMPLATE
• Pull request template, which help you a lot in reviewing code changes. Refer to https://github.com/RapidSilicon/OpenFPGA_RS/tree/master/.github/PULL_REQUEST_TEMPLATE
• Documentation. We have readthedocs setup. You can simply copy & paste my setup. Or if you feel difficult to start, I will set it up.

[komaljaved-rs]

@tangxifan Can you please specify what reports you want to be generated. Also right now whenever a benchmark is added to the master branch it is synthesized on the following tools:

• Yosys
• Vivado
• Quartus
• Diamond
• Gowin

And the update is only merged if synthesis is successful on all tools.

And secondly, by 'Ensure all the benchmarks are covered in CI' do you mean that all designs have to be synthesized once a new design is added or only the design being added. Currently all designs in the repo have manually been checked.

[tangxifan]

@komaljaved-rs

• For the report, I think for the RTL benchmarks, the most important thing is to know if it can pass the flow. You may also report other numbers, like resource counts (LUT, DSP, FF etc) as a reference, with which we know the benchmark sizes. Other performance metrics, such as logic depth, may be added further.
• For the coverage, it is o.k. that currently all the designs are manually checked. If they are already in, keep them. But our nearest milestone is to cover all of them in the CI tests. For any design which is not in the repo yet, if you want to add them, they must pass CI before getting merged.

[komaljaved-rs]

@tangxifan CI setup to synthesize all the benchmarks present in the repo is up and running. It can be triggered any time by the user or we can set it up to run monthly.

[tangxifan]

@komaljaved-rs I have not seen any log file for CI running for all the benchmarks from the Github Action page:

Since this is a RS server, I believe we can set it to run weekly. This is to ensure that any environmental changes on the RS server will not impact benchmarks

[tangxifan]

@laraibkhan-rs @komaljaved-rs I just checked your pull requests. It look like you are address this issues well following incremental changes. However, in the pull requests, you never mention which issue are being addressed. It is better to mention it so that we can keep tracking in this issue about what have been done. And we can then decide if it is enough to close this issue. Can you do it for all your previous pull requests?

Here is an example:

[komaljaved-rs]

@tangxifan The screenshot shown above is for the diamond test that is triggered on every pull request and synthesizes only added/modified test cases (if there are any). As there was no new benchmark added that's why it is giving "file exists but is empty" message but this files contains configs of added/modified test cases. I think its better to trigger it only if there is any added/modified test case during the pull request.

All the benchmarks are ruuning in this CI Setup.

I have not displayed the log files as you suggested to redirect the output to log files and display if the benchmark ran successfully or not. Can you please explain where do you want to see the the log files?

[tangxifan]

@komaljaved-rs Thanks for the updates. I understand the situation here. Here are my comments

• For log files, we should enable artifacts to be downloaded. So that people can access log files by downloading the artifacts. Please see details in https://github.com/verilog-to-routing/vtr-verilog-to-routing/pull/1857 about how to enable it
• It is o.k. to trigger the CI with a sensitive list. Only we see changes in .v/.sdc/.vhd/.sv files, it should be triggered. Github Actions support such feature. You may find details at

https://github.com/lnis-uofu/OpenFPGA/blob/d85e55aef2556459af314012c58425296a889b1f/.github/workflows/build.yml#L19-L46