Benchmarking - jsperf

[jokesterfr]

Hi everyone, thank you first for this great stuff around the Polymer project. In order to compare this native template implementation to the Handlebars templating engine (which I use in production at the office), I tried this jsperf run case : http://jsperf.com/handlebars-vs-polymer-mdv.

I've been testing it with both Firefox 21.0 and Google Chrome 29.0.1516.3 dev (javascript/webkit experimental activated in about:tags), resulting in incomparable results (x200 000 for Chrome and MDV). So I'm asking, do some people are interested here by such a comparison, and would have a closer look at my test case? Because with such a difference in the results, I'm actually wondering what I did wrong...

[rafaelw]

Thanks very much for looking at this. I'm not sure what is going on, but I'll have a look this week.

[neonstalwart]

i think i found a couple of problems with the test - one problem for sure was using raw.github.com

http://jsperf.com/handlebars-vs-polymer-mdv/7 is an updated test that looks like the results are more in line with what might be expected.

[rafaelw]

Ok, so I looked at this briefly. First: two VERY LARGE caveats:

1) Benchmarking is very hard and it's rare to encounter a benchmark which really tells you much beyond that a given code path is faster on that benchmark. This benchmark is picking something simple and I think that it doesn't say much about the relative performance in actual usage.

2) We're focusing on the semantics of MDV at the moment and have yet to start looking into perf, per se.

That being said:

The answer to why MDV is so much faster is because your test isn't really doing anything in the MDV case. Repeatedly assigning to model with the same data doesn't do anything. The databinding properly sees that the bound data hasn't changed and doesn't do anything (leaves the DOM in place).

Here's what you were probably going for: http://jsperf.com/handlebars-vs-polymer-mdv/14

If I test this in Chrome stable branch, I get that MDV is around 300x slower than handlebars. A quick look at a profile runs shows that this is because ChangeSummary is creating compiled getters for its observed data for every bound path. This could easily be optimized by using a cache, but I'm not going to do that now because, as I said, we're not yet looking at perf.

Note that If I test this is in Chrome Canary with experimental javascript enabled (which enables Object.observe), I get that MDV is about 5x faster than handlebars.

Which is really all to say that there isn't much to be said about relative performance yet. I think the Object.observe case shows MDV and handlebars will ultimately be in the same ball park, and probably vary some based on the benchmark you pick.

[rafaelw]

Note that http://jsperf.com/handlebars-vs-polymer-mdv/14 is probably the fairer version of the test since MDV and handlebars are creating an equivalent DOM fragment for each iteration.

[jokesterfr]

Thank you both for highlighting my mistakes. With your revision 14, Firefox 21 is about 30 times faster with MDV than Handlebars (compiled), still a big difference. Obviously this test cannot give a full-featured comparison, as this would require a much longer study. Whereas MDV is on its early stage, I feel it has a really good potential, and I will for sure suggest its use it in the nearby future at the office. Please provide us good semantics =)

[rafaelw]

Thanks so much for being interested in MDV! =-)