justinlubin / cobbler

Refactor programs to use library functions!
5 stars 0 forks source link

Add benchmarking pipeline #21

Closed jcbym closed 1 year ago

jcbym commented 1 year ago

Add a benchmarking pipeline to run synthesis on Jupyter notebook cells and record results in a csv file. Input cells are stored in lib/cbr_numpy/data/benchmarking/targets.ipynb, the output csv is located at lib/cbr_numpy/data/benchmarking/benchmarks.csv, and the benchmarking script can be run from lib/cbr_numpy/benchmarking.py.

Input cells must be of the format

#[NAME]
[ENV]
#synth
[BODY]

where the body ends in an expression.

Edits to support benchmarking include:

jcbym commented 1 year ago

@justinlubin Implementation details are pretty much the same as discussed on Wednesday, except I performed AST size computations in the Python script instead of in OCaml. I thought this was a bit easier since I don't have to worry about sending more things through the subprocess call

justinlubin commented 1 year ago

Hello, I fixed the line endings issue! Could you configure git to use the settings described here going forward? This is the relevant command for Windows:

git config --global core.autocrlf true

Other than that, the code looks great—thank you so much for this! I don't have any suggestions for improvement!

One thing aside from the code that I'm a bit confused about is how to actually get it to run on my computer. However, I'll merge this PR in for now so that others can build on it and assign an issue to you with some next steps that I think would be helpful for making that easier.

Thank you again!!