Pass R errors through to Conbench

voltrondata-labs / benchmarks

Language-independent Continuous Benchmarking (CB) for Apache Arrow

MIT License

10 stars 11 forks source link

Pass R errors through to Conbench #143

Closed alistaire47 closed 1 year ago

alistaire47 commented 1 year ago

Closes #134. Builds on https://github.com/voltrondata-labs/arrowbench/pull/138 to take results with an error element and pass it through to Conbench properly. Improves some tests along the way.

Tests are passing locally, but I want to run this against a local Conbench instance to make sure nothing is malformed before merging.

coveralls commented 1 year ago

Pull Request Test Coverage Report for Build 5340132227

95 of 95 (100.0%) changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.1%) to 87.38%

Files with Coverage Reduction	New Missed Lines	%
benchmarks/_benchmark.py	1	83.72%
<!--	Total:	1		-->

Totals
Change from base Build 5268118744:	0.1%
Covered Lines:	1551
Relevant Lines:	1775

💛 - Coveralls

alistaire47 commented 1 year ago

Ran against a local Conbench instance and it posts successfully:

Tracebacks are too wide to read easily, and the warnings subdict doesn't get unpacked into a subtable, but these are existing UI issues. At least the data is there so users can copy it an reformat it to see what's going on.

And we'll be able to see on Conbench when benchmarks are failing!

alistaire47 commented 1 year ago

The changes here are relatively simple and well-tested, so I'm going to merge this, but post-merge reviews are welcome; happy to do a follow-up is necessary. I'll keep an eye on buildkite to see if anything goes awry.