benchmarks 2020-12-25

[leeoniya]

i finally spent some time on a new round of benchmarks with updated libs & updated Chrome. top is previous round, bottom is current standings:

some observations:

• improvements to Chrome have boosted quite a few numbers, in some cases rather drastically. e.g. Flot perf doubled and dygraphs became significantly faster despite no lib or bench changes to either.
• apex seems to have regressed their mousemove/tootltip performance a ton
• uplot is now even faster due to simplification of the linear path-building code and completely extracting + rethinking the bands rendering logic.

let's talk about ECharts v5 -- holy shit. it went from fourth-to-last to second place. that LTTB sampling, dense array packing and whatever other black magic is going on made it fly. what's worrying about this is that this lib is almost 1MB (30x larger than uPlot), yet the initial parse/compilation does not seem to have that large an effect, even when reloading with caching disabled in devtools. since these measurements are taken from close-to-best runs, i think that there is still a JIT cache that persists, and chrome simply uses the content hash to revive the prior compiled bytecode. i tried to record a few runs from an about:blank page and it seemed to confirm that there is some caching going on that cannot be disabled from devtools, though it is not as drastic as i expected -- a cold boot is about 360ms vs 435ms cumulative - a 75ms difference, which is less than the difference when running with what i assume is a warm JIT cache. i honestly have no idea where exactly parsing and JITing an extra 900KB of JS is accounted for, and it scares me that i'm doing something wrong or devtools is lying to me. if someone has any idea what's going on, please clue me in! either way, the results are nothing short of astounding, so massive props to @pissang.

refs: https://github.com/apache/incubator-echarts/pull/13314, https://github.com/apache/incubator-echarts/pull/13339, https://github.com/leeoniya/uPlot/issues/317

cc @benmccann @ryantxu

[pissang]

From my observation. The first run without cache is about 10x slower than fully JIT code, which is warmed up by running the same code several times. And the variance of the first run is much larger. I guess the time of compiling to JIT code leads to this large variance.

[kurkle]

Looked at the full sized plot of the results and noticed the labels don't align (many are missing). https://leeoniya.github.io/uPlot/demos/multi-bars.html

Did not investigate any further if this is an issue with the demo or uPlot itself (and hence commenting here and not filing a bug)

another note about the plot, it would be logical to have all the ms bars on same scale:

[leeoniya]

Looked at the full sized plot of the results and noticed the labels don't align (many are missing).

thanks, fixed in https://github.com/leeoniya/uPlot/commit/7990578ad12e9582cb06cd48243dc1853fdcdab5. that pesky devicePixelRatio!

another note about the plot, it would be logical to have all the ms bars on same scale:

the metrics are different enough that i think different scales are warranted -- "ms of initial render" is not really the same as "ms of toggle", or "ms of interaction". putting them on the same scale may falsely exaggerate or compress one type of metric at the expense of another, since their profiles are so different.

[BorisKozo]

Hi, could you please change line 38 in HighCharts example to let date = packed[i] * 60 * 1000; (remove the new Date() part) It will result in the same chart that, on my machine, runs twice as fast for some reason. Thanks.

[leeoniya]

@BorisKozo yes, it looks like you're right.

i think i tried this when adding HighCharts initially and maybe it did not work?

is this a new capability? i still don't see documentation or examples using it anywhere:

https://api.highcharts.com/highcharts/series.line.data https://www.highcharts.com/docs/advanced-chart-features/boost-module

i will update the bench later today. thanks for the heads up!

[leeoniya]

ok, results are updated (moved up a rank). i also found how to disable animations, which got a good improvement on toggle performance and helped a bit on interaction (mousemove).

[BorisKozo]

Thanks!

[danr]

I would be interested in seeing the "transpose" of this plot: one plot for each metric (file size, time measurement, heap size) with the different tools next to each other in that metric's subplot. As the plot is right now it is not immediate which tool is best on a given metric.

[leeoniya]

transpose would mean 15 bars & colors per group and 6 groups :)

instead it's probably best to add a sort-by selector with each of the 6 metrics.

[danr]

I think it would be 6 interesting groups!

[leeoniya]

here you go: