search

h2oai / db-benchmark

reproducible benchmark of database-like ops
https://h2oai.github.io/db-benchmark
Mozilla Public License 2.0
326 stars 88 forks source link

Add version of Julia, R and Python to main page #66

Closed mattdowle closed 5 years ago

mattdowle commented 5 years ago

It's great the development versions of the packages are used and stated, but also what about the version of the language itself. The feedback today had Julia in mind since it is most in development (is db-bench using Julia-dev?), not so much R and Python. Aside: if we could switch to use R-devel that would be a useful service to R-core, too. We've seen before that R-devel changes do impact data.table performance so it would be better to know as soon as those changes are made. If we saw a drop in performance one day we wouldn't be sure if it was due to a change in the package or the change in the language, though, so that would be an extra complexity to deal with somehow; e.g. upgrade the language once a week without any product update.

jangorecki commented 5 years ago

Agree on testing on R-devel, but most likely we would miss impact of it, unless it would be really significant. We would need proper reports on detecting performance including language version, as of now we don't even have mechanism to warn about performance regression within a single solution over different versions/git revisions. Lets narrow this issue to mention languages version on report.

mattdowle commented 5 years ago

I see the language versions on report now. Great. image There's no need for that sentence starting 'Listed solutions...' as it's already clear from the heading. Just remove it please. If you keep it then 'where' should be 'were'.