search

VROOM-Project / vroom-scripts

BSD 2-Clause "Simplified" License
35 stars 23 forks source link

Results of Benchmarks #14

Closed bkj closed 4 years ago

bkj commented 4 years ago

Are you able to upload the results of these benchmarking scripts here?

I see it should be easy to run your scripts, but it'd be super helpful if you could upload the results here as well so I can validate my results.

Thanks!

jcoupey commented 4 years ago

The results on the various benchmarks consist of a somewhat huge number of files (we don't even store the input files as json here). Beyond the fact that it would bloat the repo, I think it would somehow defeat the whole purpose of making available a fully reproducible toolchain.

On top of that, comparing actual solutions one by one is a tedious process and we already have global indicators and gaps to best known solutions reported by compare_to_BKS.py (automatically triggered by run.sh). If you're just interested in validating the results you get from the toolchain, it may be enough to just compare these with the results reported at https://github.com/VROOM-Project/vroom/wiki/Benchmarks.

Side notes: results may obviously vary in computing time, but also somehow in quality due to https://github.com/VROOM-Project/vroom/issues/145.

jcoupey commented 4 years ago

Closing as stale.