search

h2oai / db-benchmark

reproducible benchmark of database-like ops
https://h2oai.github.io/db-benchmark
Mozilla Public License 2.0
324 stars 87 forks source link

Order results within each test, perhaps. #34

Closed mattdowle closed 5 years ago

mattdowle commented 6 years ago

Each software is already colored, and the syntax is next to each bar, so if they are ordered by time within each test it might be easier to more quickly compare and it won't be difficult to identify which software is which. Question is, what to order by: first run time, or average of both, or total of both runs. As more packages are added, ordering within test might become more needed. As always, providing a toggle to let the viewer choose the sort method that is appropriate for them would be ideal, but more work to achieve of course.

jangorecki commented 6 years ago

currently solutions are in alphabetical order

st-pasha commented 5 years ago

It is useful to have a stable order between different questions. On the other hand, alphabetical is too arbitrary. We could order the packages by the geometric mean of their timings across all questions -- this would have almost the same result as Matt's suggestion, while avoiding the packages switching their order in different questions.

mattdowle commented 5 years ago

It is useful to have a stable order between different questions.

Why precisely? For each question I think most people are interested in which solution is fastest. There is a table at the top of the image (the legend) which could be sorted by time (total or mean) of the 5 questions. There are no y-axis labels currently: if the software names could be added there, it wouldn't be necessary to know the colors or read the syntax to know which was which: it would be easier. But apparently there's a width constraint of the page to overcome. Another reason for sorting by time within each question is then it's even easier to see the solutions that failed: they would be at the bottom after the longest running (but at least working) solution.

st-pasha commented 5 years ago

Why precisely?

Otherwise it will confuse colorblind people, and it also goes against the advice of UX experts (https://ux.stackexchange.com/questions/109019/should-a-bar-chart-redraw-categories-in-size-order-or-stay-fixed-position)

mattdowle commented 5 years ago

@st-pasha That's why I wrote above :

There are no y-axis labels currently: if the software names could be added there, it wouldn't be necessary to know the colors or read the syntax to know which was which: it would be easier. But apparently there's a width constraint of the page to overcome.

Where "to overcome" meant @jangorecki please overcome :-) and add product names to y-axis.