criteria for performance graph

[vezeli]

I couldn't help but notice the final results of the performance graph look as if they are sorted according to the performance of each programming language but than taking a closer look shows that Python seems to be outperforming Matlab and Mathematica, also it is disputable if Rust and LuaJIT are slightly outperforming Julia or not.

Do you think maybe listing the results alphabetically or clearly specifying an objective criteria used for ordering the results would be better than keeping the results in this form? As I see it, this way of presenting might lead to misinterpreting the results of the study...

[waldyrious]

The sorting method is described in comments in the code:

https://github.com/JuliaLang/Microbenchmarks/blob/d97a03050052faf22308cfdb60c8066431b6e6da/bin/table.jl#L53-L62

but I agree it would be nice if this was exposed in the final graph image, or at least in a caption.

[vezeli]

I see. If that is the case one could also consider showing the results of the geometric means next to the names of the programming languages for the sake of clarity and transparency.

[StefanKarpinski]

Yes, plotting the geometric mean as an extra plot point seems like a good idea 👍

[johnfgibson]

@waldyrious: that table.j code is no longer in use. It produced the HTML text for a table of normalized timings, language versus benchark, which is no longer shown on the benchamrks page. The sorting is now done in the IJulia notebook linked to towards the end of the benchmarks page: https://nbviewer.jupyter.org/url/julialang.org/benchmarks/benchmarks.ipynb. The geometric means for each language are listed at the end of this notebook.

Agreed that the geometric mean data should be provided on the benchmarks page and that the text should explain the ordering, particularly that it's C, then Julia, then everything else sorted by geometric mean. I'm not sure adding another dot to the plots is the right way --it's already crowded with many overlapping dots. Maybe geometric mean plotted as a black dot would work, or geometric mean / 10. I'll play around with it when updating benchmarks for Julia 1.2.

[waldyrious]

that table.j code is no longer in use. It produced the HTML text for a table of normalized timings, language versus benchark, which is no longer shown on the benchamrks page.

Should we remove the table.jl file, then? Or at least add a comment at the start of the file mentioning that it's deprecated?

The sorting is now done in the IJulia notebook linked to towards the end of the benchmarks page: nbviewer.jupyter.org/url/julialang.org/benchmarks/benchmarks.ipynb. The geometric means for each language are listed at the end of this notebook.

Would it make sense to include that ipynb file in this repo?

Maybe geometric mean plotted as a black dot would work

Perhaps using a different marker (say, a thin horizontal line) would make it easier to stand out among the other dots, and it wouldn't even need to be a strong color (it could be gray instead of black, for example).