voltrondata-labs / arrowbench

R package for benchmarking
Other
13 stars 9 forks source link

Modify returned benchmark scripts to remove abstraction #100

Open boshek opened 2 years ago

boshek commented 2 years ago

Currently arrowbench attaches the script used to run the benchmark to the returned json. However that script uses some functions in arrowbench. As much a possible, it would be good to modify that to use the actual functions being benchmarked.

Eg.: for the write_csv benchmark we have a line like: get_csv_writer(writer) where writer is "arrow" . Ideally that should be expressed as write_csv_arrow

alistaire47 commented 2 years ago

I was thinking yesterday that we should move all the script writing to a list of expressions instead of strings; rlang makes working with language objects manageable, and it would validate syntax (which would be nice) and eventually allow us to do this kind of substitution if we set it up right.

Here's an example where I worked with code as expressions in rlang, though that's probably more abstract than we need; we can straight up use quote()/expr() for a lot

nealrichardson commented 2 years ago

What about an entirely different approach? I wonder what it would look like to use a templating framework (glue might even be enough) to generate the scripts that we run, scripts that are fully self-contained and that don't depend on arrowbench even. There would be no abstraction to remove from the scripts anymore. arrowbench would just contain code for parametrizing the templates, running the scripts, and processing the results.