voltrondata-labs / arrowbench

R package for benchmarking
Other
13 stars 9 forks source link

Record failure + write to json file #85

Closed jonkeane closed 1 year ago

jonkeane commented 2 years ago

Currently, when a benchmark fails, we do not write anything to disk but now that conbench can accept errors, we probably should be writing these results to disk + marking them as errors (and including the stack trace from the subprocess in the output.

This will (likely) require changes to https://github.com/ursacomputing/arrowbench/blob/4022c6e37a8aa2eebd579a8d710ee5e05c1c7372/R/run.R#L305-L312 as well as https://github.com/ursacomputing/benchmarks/blob/b252842cc629fdea4eb8eec2b9db9f1eb7324040/benchmarks/_benchmark.py#L197-L204 where we will probably need to add a step where we read in the result + mark it as errored (possibly in: https://github.com/ursacomputing/benchmarks/blob/b252842cc629fdea4eb8eec2b9db9f1eb7324040/benchmarks/_benchmark.py#L220-L229)

alistaire47 commented 1 year ago

This is kind of done in #127, but not as-written—errors are getting reported to Conbench now (if run through arrowbench instead of labs/benchmarks), but not written out as files, as the erroring result is already created in the parent process, so we don't need a file. We could write one out, but I'm not sure if we want to cache errors anyway.

jonkeane commented 1 year ago

nods yeah I'm fine to close this. The original "write a file" was mostly a shortcut to when we were running with conbench|benchmarks where that was looking for a JSON somewhere. We've solved the spirit of this in #127 so I'm happy to close. We can reopen or make a new issue if we need