perf benchmark: ractive blows hot and cold

[sabob]

In a nonsensical benchmark described here: http://www.stefankrause.net/wp/?p=191

they measure 1000 rows created, updated and a row removal.

Interestingly Ractive is the fastest at updating 1000 rows and slowest at removing a row ;-)

Thoughts?

[evs-chris]

I haven't looked in-depth, but I do know that edge was way faster than 0.7.3 on several of our internal rendering benchmarks last time I checked.

[fskreuz]

Ractive.js is quite extreme. It’s the fastest when all rows are updated and the slowest in the “remove row” benchmark.

Possibly attributed to the fact that Ractive "surgically" updates the DOM. It tries its best to recycle DOM and only update what's necessary when necessary in a way that is necessary. An earlier wave of benchmarks were done using DB monster. @Rich-Harris did one for Ractive here https://github.com/Rich-Harris/ractive-dbmonster

Why is Ractive so fast?

Ractive uses similar techniques to Glimmer - it parses the template into a structure such that it's very easy to identify which DOM nodes need to be updated when data changes. This is different to how React (for example) handles DOM updates, which involves re-rendering everything and then running a diff.

Here's some demos thrown in during that period

• Ember https://dbmonster.firebaseapp.com/
• React http://run.plnkr.co/plunks/Wwgjjpl9NHMO5Nd1TUyN/
• Ractive http://www.rich-harris.co.uk/ractive-dbmonster/
• Paperclip (afaik, the fastest) http://paperclip-dbmonster.herokuapp.com/

On the other hand, benchmark implementation isn't usually "equal". There's a possibility that some are optimized for the task while some aren't. For instance, Ractive's rows are defined inline to the main component, while React's are defined in an entirely different component. What if Ractive used a component for the rows? What if React used inline rows? It could make the difference.

I'd say the DBMonster benchmarks were closer to "fair" as the framework authors themselves wrote the apps towards the same goal using a common data source. They could have thrown in everything to make their version really fast.

[sabob]

Running the test locally in edge shows edge is quite a bit faster than 0.7.3, but only in the things 0.7.3 is already good at, updates ;-)

Initial rendering of 1000 rows and deleting a row is where ractive doesn't shine so brightly.

Although nonsensical, sometimes these micro benchmarks highlights low hanging fruit. Whether it is worth pursuing those gains is another matter of course.

Wrt to the benchmark, I can understand ractive taking a bit of time in the create 1000 rows" test, it probably does more work upfront setting up the virtual dom etc. For example, running the "create 1000 rows" test on my machine takes 1000ms, on the first run. On the second run it takes 130ms, since it reuses the existing dom that was created I assume.

I am surprised with the "delete 1 row" test though. That takes 500ms. Maybe their ractive test is borked? They essentially do this to remove a row:

Template:

 {{#each store.data}}
    <tr>
        <td><a on-click="remove(id)"></a></td>
    </tr>
{{/each}}

Code:

new Ractive({
delete: function(id) {
    var index = findIndex(id);
 startMeasure();
  ractive.splice('store.data', index, 1);
  endMeasure();
   ...
}

That is normally how I delete rows as well ;-)

Not sure why it would take so long to remove a row? Doesn't matter if you delete the first or last row.

[evs-chris]

I just tried it with edge, and the initial render is a bit slow still. The remove is down to 1ms, though, and every other benchmark is ~30ms. Here's what I get on my machine:

test	0.7.3	edge
create	850ms, 152ms, 131ms, 125ms, 131ms	903ms, 108ms, 95ms, 96ms, 94ms
add 10	80ms, 84ms, 78ms, 80ms, 82ms	27ms, 39ms, 30ms, 33ms, 36ms
update 10	71ms, 74ms, 72ms, 70ms, 72ms	21ms, 21ms, 21ms, 22ms, 23ms
remove first	806ms, 359ms, 353ms, 304ms, 311ms	1ms, 1ms, 1ms, 1ms, 1ms
remove last	22ms, 17ms, 18ms, 18ms, 19ms	1ms, 1ms, 1ms, 1ms, 1ms
remove middle	189ms, 191ms, 193ms, 189ms, 189ms	1ms, 1ms, 1ms, 1ms, 1ms

The times each represent a click, with the first being the first click. 0.7.3 remove performance varies wildly from start and end because it has to mark everything after the altered element due to the ways its caching works. @martypdx really nailed splicing with the model approach in edge.

So the only place 0.7.3 wins is in initial render. After that, edge wins all around, sometimes quite resoundingly. My current Ractive play issue is conditional directives and attributes, but once I get that settled, I'll look into initial render performance. It's been biting me a little in one of my projects that can use edge anyhow.

[evs-chris]

Hmmm... I apparently wan't paying close enough attention to what was happening, because the remove doesn't actually remove with edge. I suspect the slightly wonky way that the data manipulation is done in the test is the issue.

[sabob]

Nods, in the file 'main.es6.js' they do this:

this.data.splice(idx, 1);

To make it work with edge I changed it to:

ractive.splice('store.data', idx, 1);

[sabob]

Btw, I left out an important bit when I posted the example template above. They have an expression on the table row:

{{#each store.data}}
   <tr class="{{ selected == id ? 'danger' : '' }}">
        <td><a on-click="remove(id)"></a></td>
    </tr>
{{/each}}

Removing that expression and ractive removes row in 170ms, instead of 500ms. Adding a second expression in the repeated section and remove time goes up to 1400ms.

[evs-chris]

A quick profile shows a ton of time in resolveAmbiguousReference. Making all references unambiguous (.id vs id) cuts the time to a fifth of what it was on my machine.

[sabob]

Seems Ractive rebinds all expressions when the array is sliced? If the expressions referenced the row index, I can understand rebinding is needed but here it seems unnecessary? Ractive doesn't need to do anything really, just zap the dom?

[heavyk]

I think there should be a note somewhere about resolveAmbiguousReference. it's really important to understand that

in our projects, I updated all {{foo}} to {{.foo}} or {{~/foo}} depending on the context. now I don't trace any more ambiguous references, so that sped everything up 2-8x

[sabob]

From what I can gleam so far is that resolveAmbiguousReference eventually calls:

• RootModel.prototype.get, which calls
• RootModel.prototype.extendChildren

extendChildren then iterates all the model's computations, checks if the computation is an expression before invoking a callback function, if it is not.

There are 1000 computations in the model at this stage and since there are 1000 rows, we loop over the model computations, 1 000 000 times. For this benchmark, all the computations are in fact expressions and we end up looping 1 000 000 times for no reason, except to chew up time ;-)

Seems like there is some optimizations we can do in repeating sections, some caching maybe?

As @evs-chris noted, using absolute references speeds things up quite a bit as it skips this extendChildren bit, however using {{foo}} is much nicer than {{.foo}}

[heavyk]

@sabob true, but if you had something like this:

Ractive.extend({
  data: {
    list: [{one: 1, two: 2}, {two: 11}, {one: 1}],
    one: "yay"
  },
  template: `
    {{#each ~/list}}
      {{one}} {{.two}}\n
    {{/each}}
  `
})

then the output would be:

1 2
yay 11
1

see, the '.' tells to look in only in the topmost scope for that name. it's by design. it is annoying, I know - but once you get it, it's powerful because you reduce the number of lookups.

perhaps it could be cool to flip it around and say that {{foo}} and {{.foo}} (for sake of backward compatibility) resolve only in the top scope, and {{?foo}} or {{foo?}} resolves the ambiguous reference

EDIT: lol wrong button. fixed. EDIT2: got my explanation backwards from the code.

[woodlandhunter]

@heavyk you may have left a "not" out of your explanation.

[heavyk]

did I? I didn't test the code... which not? I don't understand.

I'll make a fiddle. one moment

[heavyk]

works as expected: http://jsfiddle.net/bxaeqb5a/

[sabob]

@heavyk thanks for the example. My comment on "{{foo}} nicer than {{.foo}}" is not that the feature isn't a good one, but rather I don't want to change all my references to {{.foo}} because of performance.

[sabob]

@evs-chris great work on #2372.!

Here are some numbers, before and after:

Test	Before #2372	After
Create 1000 rows	1 485ms	770ms
Add 10	39ms	30ms
Update 1000 rows	126ms	124ms
Update every 10	66ms	66ms
Remove first row	1390ms	260ms
Select row	26ms	26ms

There is also a follow up to the benchmark as other projects are also busy optimizing ;-) http://www.stefankrause.net/wp/?p=283

I'll do some more profiling, mostly to learn about JS optimizations. Is Chrome profiler the best available at the moment or it subjective?

[guilhermeaiolfi]

@sabob can you run the whole bechmark (all frameworks) to see how ractive compares to others now? I tried here, but npm is not in a good mood in this machine I'm using.

[sabob]

@guilhermeaiolfi I cannot get all the tests to run either, probably Windows issues. I can run Angular, Angular2, Ember, Ractive and React. I can sort of deduce which one is the fastest in each category:

Test	Ractive before #2372	Ractive after	Best
Create 1000 rows	1 485ms	770ms	320ms Vidom (Angular 2 is 404ms but vidom seems to be 20% faster)
Add 10	30ms	20ms	12ms Angular2
Update 1000 rows	126ms	124ms	124ms Ractive
Update every 10	66ms	66ms	25ms Angular 2
Remove first row	1390ms	260ms	141ms Angular 1
Select row	26ms	26ms	6ms Angular 2
Add another 10	30ms	30ms	30ms Ractive (Angular2 is 100ms.)

The test "Add another 10 rows" is my own benchmark, where I add 1000 rows and then another 10 on top of that. I cannot determine which one is best.

[woodlandhunter]

@heavyk You said:

see, the '.' tells to look in any scopes below it for the same name

.foo restricts to the current context, but hopefully that is already clear to anyone else reading the thread.

http://docs.ractivejs.org/edge/references#restricted-references

[sabob]

Setting "modifyArrays" to false, shaves off a couple of ms:

Test	Default	ModifiyArrays: false
Select	26ms	18ms

[sabob]

Ouch, I made a mistake in my test. For the Ractive tests I accidentally added an expression which the other frameworks didn't have. Here are the revisited numbers:

Test	Ractive before #2372	Ractive after	Best
Create 1000 rows	1 080ms	740ms	320ms Vidom (Angular 2 is 404ms but vidom seems to be 20% faster)
Add 10	17ms	15ms	12ms Angular2
Update 1000 rows	126ms	124ms	124ms Ractive
Update every 10	26ms	26ms	25ms Angular 2/Ractive Close enough to be a tie
Remove first row	750ms	230ms	141ms Angular 1
Select row	15ms	15ms	6ms Angular 2 (Ractive with modifyArray false: 11ms)
Add another 10	27ms	27ms	27ms Ractive (Angular2 is 100ms.)

[heavyk]

@woodlandhunter you're right. I edited my post to say it properly. I don't really remember what happened that night; I might have been stoned at the time :/

@sabob I didn't expect modifyArray: false to have any effect on select times. does anyone know why? I do a lot of array operations in our projects so I imagine that I'll be doing some benchmarks tonight. cheers! thanks for the tip :)

[sabob]

@heavyk I need to do a PR for the modifyArray speedup. As for why it would speed things up by a couple of ms is:

for the select benchmark, ractive loops every item in the array (1000 items) and checks if it's ID is equal to the selected one. If it is, it it highlights the row. So for this select operation, ractive uses an O(n) algorithm. The more items in the array the longer select takes. For every loop ractive performs certain operations, one of which is to adapt the value based the registered adaptors. modifyArrays is a default adaptor registered by ractive.

The Model.prototype.adapt method has this line:

const keypath = this.getKeypath();

which in a tight loop chews a couple of ms. By setting modifyArrays: false the adapt method guts should be bypassed, however as it stands the line above line is still executed, since the adapt method doesn't bail immediately if there are no adaptors. We could change adapt to:

adapt () {
    const adaptors = this.root.adaptors;
    const value = this.value;
    const len = adaptors.length;
    if (len === 0) return;
    // slow code below
    const keypath = this.getKeypath();
    ...

However this is splitting hairs. Probably not really worth adding this tweak.

[heavyk]

well explained. that sounds reasonable to me. I didn't know it did that...

I think it's worth it and I'd definitely put up a PR for the early return with no adaptors. even though it's likely that someone is going to have at least one adaptor installed, at least that gives the option to someone for a performance boost in exchange for magic. for me, it's little effort to change the way I use my arrays and would gladly make the change.

[tomekmarchi]

I have noticed by writing a standard for loop and replacing .forEach not only compresses the library further (as when it is compressed over head is one letter instead of .foreach e()) & is much faster. I will fork Ractive later today with a few changes like this. For example instead of doing .length use a function this will compress it and performance will remain virtually the same. However the .length change is more for the compression gains. Also removing this old isArray method with the native isArray seems to be a better choice for the compression gains off the rip. I'm looking to make changes to the library that focuses on all MODERN browsers. Will provided findings and results as the days go by. Cheers!

[dagnelies]

Just to add my 2 cents, I personally favor good code readability and good browser support. ...and consider gaining a few nanoseconds counter-productive if it hinders either of these aspects. Ractive's performance is more than enough. In almost all cases, the browser will spend most of the time in the layout engine or other places anyway. Please keep that in mind. Thanks.

[evs-chris]

Edge is now in the latest version of the benchmark results.. We're not exactly leading the pack due to the complexity of creating and removing massive amounts of DOM, but we're doing decently well at an overall 1.69x pure js, which is toward the front of the middle of the pack.

I've looked into speeding up the creation and removal time, but I can't come up with a way to do it nicely with how Ractive works, and honestly, I'm not sure that it's worth it for this particular case. It's probably not the best idea to throw 10,000 rows at a user. Even 1,000 is probably pushing it. If you do need to do stuff like that, don't get bootstrap css anywhere near it, cause it apparently murders performance.

So I'm going to close this. Edge, which will hopefully be 0.8.0 very soon, is anywhere from 30 to 500% faster than 0.7.3 on the tests in this benchmark that are heavy.