search

cucapra / calyx-resource-eval

Resource Usage Evaluation for Calyx (& its Frontends)
0 stars 0 forks source link

First attempt at adding SDN #18

Closed anshumanmohan closed 2 months ago

anshumanmohan commented 2 months ago

This PR adds SDN to the test suite. The README shows you which script to run, and the results go into sdn/results in individual JSON files.

Questions for @calebmkim:

  1. I copied over the Polybench directory and modified it, so I have a synthesis_scripts directory but I'm not sure if I need it.
  2. I don't think I am generating cycle counts. Thoughts on that? Should that just be a different set of four fud commands?
  3. I am not expect-testing yet, although the .expect files are checked in. Again, thoughts on that?

I have used the latest .futil files from the current version of Calyx. The numbers have changed (for the better) since the paper but, in my case, I can easily check out the relevant .futil code from a few months ago and get back the same numbers as the paper. However, if the cycle counts (in particular, the C/push number) continues to follow the trend that we showed in the paper, we could perhaps just stick to these latest numbers and say that the overall subjective points are still being made. I can't make a decision on this until we start generating cycle counts (see above).

calebmkim commented 2 months ago
  1. I copied over the Polybench directory and modified it, so I have a synthesis_scripts directory but I'm not sure if I need it.'

Yeah I don't think you need it. In fact, I don't think the dahlia-polybench folder needs it. I think I should delete it.

  1. I don't think I am generating cycle counts. Thoughts on that? Should that just be a different set of four fud commands?

Yeah, you can generate cycle counts using jq. It's this command right here: https://github.com/cucapra/calyx-resource-eval/blob/7d1fbdefc4ed6779fb483ce5f2a0b62b77edee16/dahlia-polybench/scripts/polybench_calyx.sh#L15C62-L15C157. Basically, you run simulation through verilator, and then use jq on the output to just get the "cycles" entry of the resulting json.

  1. I am not expect-testing yet, although the .expect files are checked in. Again, thoughts on that?

Yeah, one option is to just have a separate script that checks the simulation results match the .expect file. It's right here for the dahlia-polybench stuff: https://github.com/cucapra/calyx-resource-eval/blob/main/dahlia-polybench/scripts/polybench_calyx_check.sh.

calebmkim commented 2 months ago

If you do not object, I can take a shot at adding the cycle counts on this PR. Since you've provided the .futil files already it should be pretty straightforward.

anshumanmohan commented 2 months ago

Thanks!! No objection at all! If you have time, please do all that you need to and get us all the way home! I will not have time to work on this until tomorrow am sadly.

anshumanmohan commented 2 months ago

Gorgeous, thanks so much @calebmkim! First thing tomorrow I will triple check that the qualitative point still holds. Then I will help you craft the paragraph that explains the difference for this and other benchmarks.