seed-rs / seed

A Rust framework for creating web apps
MIT License
3.8k stars 153 forks source link

JS Framework Benchmark performance #516

Open rebo opened 4 years ago

rebo commented 4 years ago

JS Framework Benchmark

The current de facto standard benchmark for measuring the performance of front end apps seems to be https://github.com/krausest/js-framework-benchmark.

This is a fairly straightforward benchmark covering a number of actions a front-end framework would typically perform.

The key advantage of this benchmark is the large number of current frameworks represented as well as an easy to run benchmark runner.

This benchmarks renders a long list of three random words and completes a number of operations. Typically the list is 1000 rows long.

image

The Seed benchmark on the site is an old one from Seed 0.6.0. The Seed code is for this project is representative of well written seed code therefore I have analysed against this.

The purpose of this issue is use the benchmark to:

a) Understand the performance profile of existing seed. b) Implement some ideas (perhaps from other frameworks) to improve this profile.

In this first post I will discuss some specific bench mark tests in order to build understanding for (a) above. In the next post I will show some possible improvements.

Existing results

Here are the results on the benchmark site for seed along side the top 3 ranked frameworks (typically vanillajs is one of them) as well as Elm and React.

image

as you can see Seed is competitive with React/Elm for some of these benchmarks but for several it is substantially slower. Specifically partial update select_row and swap rows have very poor comparative performance.

I believe the reason for this is primarily down to Seed's rendering architecture. (However something I am unclear on is why Elm does not have the same issues as I understand Elm and Seed have similar rendering architectures).

To be clear what I mean by "rendering" consists of the entirety of executing the update function, view function, as well as the virtual-dom reconciliation and browser api interaction that occurs afterwards I..e everything that has to happen in order for an effect to appear on the screen.

How Seed renders

What happens when Seed renders is that after the update function has mutated the app Model, the top level view function is called, this in turn calls many other view functions until the entirety of virtual dom is built from scratch. The virtual dom being a tree of Node<Msg>

Once this Node<Msg> tree has been built it is reconciled (diffed) against the old virtual dom and this results in actions to update the real browser dom via the browser api.

This has the effect that as your view tree becomes bigger and bigger the app becomes slower to both build the dom tree and reconcile the dom tree with the old stored dom tree. This cost is paid no matter how small an update you wish to process.

Now as long as your view can be built and reconcillled within 16ms you are going to effectively have 60fps response anyway. So often this is not going to be a problem. Usually this is actually the case and these issues wont rear their ugly head. That said it is still worth considering these issues and possibly improve this situation.

Small change - Big Cost

Because any change causes the rebuilding of the entire Node tree, this is most inefficient when you have a tiny change that might only affect a small part of the browser dom.

This "small change-big-cost" can be seen in the case of the partial update benchmark. What this does is simply add "!!!" to the end of every 10th row's label. The current benchmark seed app updates every 10th row's label String field by appending "!!!". The view function then rebuilds the entire view from scratch, iterating over 1000 rows and building each one via view_row().

This is inefficient because 99% of the rows are unchanged however they are all "rebuilt" because that is how the view function works. There is an additional cost in reconciliation, again 99% of the virtual dom tree has not changed however each node has to be compared to see if it actually has.

This results in a render time of 525ms. Compared to 133ms for vanilla seed and 209ms for react.

An even more extreme effect of this can be seen in "select row". This simply adds the class danger to the clicked on row, and removes the class from any existing row.

The seed time for this (at 16x slowdown) is 862ms compared to 22ms for vanillajs and 113ms for react. Interestingly Elm is takes41ms.

So what is vanillaj-s and react doing in order to prevent the larger 862ms cost?

vanilla-js

Vanilla-js simply mutates the dom nodes directly, it is not part of a framework and therefore does not need to represent a virtual dom.

React

Although react has a virtual dom, it is able to save approximately 85% of the work by using the concept of components. This is a concept that seed does not directly have or utilise. Basically a React's virtual dom is a hierarchy of components, these can be selectively rebuilt on the proviso that any child components are also rebuilt.

This means that in the equivalent react app, if a row is a component then only that row needs to be rebuilt on a change that affects that row because it has no children. This means that the entire virtual dom does not need to be reconstructed and therefore reconciliation cost can similarly be truncated.

The reason why react is not as good as vanilla JS is because it still has to rebuild the entire row and do some reconcillation.

Elm

I do not know why elm is so performant here (someone with knowledge of how Elm works might be able to help?). I thought it followed a similar model to Seed in that the entireity of the virtual dom is rebuilt each update. However this cannot be the case because 41ms (at 16x slowdown) is vastly quicker than Elm takes to complete the create rows test in the first place.

In the next post I will write up some advances in these areas.

rebo commented 4 years ago

Given the above I am looking for efficiencies that can be made that have the following characteristics

1) Can be applied generally to all apps. 2) Don't require massive structural changes to either Seed or the App's architecture. 3) Continued use of a declarative way of writing seed apps.

Some suggest that removal of the virtual dom is the way forward, certainly many frameworks have success in this area (Svelte/Solid/Imba) and some in the rust world (Valorie). However having implemented a dom-less Seed it enforces a huge amount of architecting and re-engineering that I am still unconvinced it is viable more generally and is better left to other frameworks being built from scratch.

That said there are things that we can learn from the 'domless' approach and we can actually benefit directly even though a virtual dom is still being used.

What a domless architecture does

A domless architecture has 2 phases, a paint phase and an update phase. The paint phase is pretty much identical to any framework (including seed), it builds the dom view in the browser based on a declarative representation.

Where the architecture is different is in the update phase, events/mutations to state directly cause browser dom effects and directly update the view on screen.

The advantage of this is clear, if state that determines the class of an element has changed then this will only change the class attribute in the browser and do nothing else. There is no problem of inefficiently building a virtual dom or inefficiently reconciling because these are not created to begin with.

How can Seed learn from this?

Even though Seed does have a virtual dom we can actually apply the same principle. We can render as per the virtual dom however on specific updates we can still reactively affect the browser directly.

In order to do this we need a way of determining that a specific state has changed which both is representable in the virtual dom view function as well as scheduling update code if that state changes in the future.

We can do this using Seed hooks.

Updating the Seed benchmark.

Because we want to represent reactive labels as well as selection state we change the Row definition from this:

struct Row {
    id: ID,
    label: String,
}

to this:

struct Row {
    id: ID,
    selected: StateAccess<bool>,
    label: StateAccess<String>,
}

This creates two state variables that hold a String and a bool. The reason we are using Seed hook state variables is that they can be observed and reacted to which will be used to directly update the browser dom.

The label

In order to react to changes to the label directly we use the reactive() method on the StateAccess value. in the row view we change from this:

a![C!["lbl"], &row.label],

to this:

a![C!["lbl"], row.label.reactive()],

This always returns the state of the label for the virtual dom, however it also schedules an update to the relevant element whenever label is updated.

Updating the label is handled as per normal seed in the update function. We change from this:

Msg::Update(step) => {
            for position in (0..model.rows.len()).step_by(step) {
                model.rows.get_mut(position).unwrap()
                .label += " !!!";
            }
        },

To this:

Msg::Update(step) => {
            for position in (0..model.rows.len()).step_by(step) {
                model
                    .rows.get(position).unwrap()
                    .label.update(|label| *label += " !!!");
            }
            orders.skip()
        }

This simply changes from mutating label directly to updating it via the state's update method.

The result of this is as follows. On initial render the vdom is created as normal, on subsequent updates it will directly modify the dom node contents with the contents of label.

The class selection

In order to react to changes to the selected row directly we use the reactive_class() method on the StateAccess value. in the row view we change from this:

tr![
    attrs!{At::Class => if is_selected { AtValue::Some("danger".into()) } else { 
    AtValue::Ignored } },
     ...
tr![
    row_selected(row.selected).reactive_class(),

This uses a row_selected() reaction in order to render a class in response to selection changes as per this function:


#[reaction]
pub fn row_selected(selected_state: StateAccess<bool>) -> Reaction<(At, AtValue)> {
    if selected_state.observe() == true {
        (At::Class, AtValue::Some("danger".into()))
    } else {
        (At::Class, AtValue::Ignored)
    }
}

if the selected state is true, it renders a "danger" class attribute snippet.

Finally the update function is changed to reflect the selected state being stored on the row as well as the model.

We change from this :

    Msg::Select(id) => model.selected = Some(id),

to this:

 if let Some(old_id) = model.selected {
                model.rows[Row::position(old_id, &model.rows)]
                    .selected
                    .set(false);
            }
            model.rows[Row::position(id, &model.rows)]
                .selected
                .set(true);
            model.selected = Some(id);

Now this is somewhat more complex because we are setting the selected state on the row directly in addition to storing it on the model. We de-select the state on the old row and re-select it on the new row.

The effect of these changes.

These relatively small changes to the seed app leveraging the reactive features of the Seed hook state variables result in the following improved performance:

image

I have also made similar improvements to the swap_rows bench. As you can see partial update and select rows are now nearly as performant as vanilla-js. And much more performant than either React or Elm. There is some small additional cost in initial creation of the rows.

I will update this issue as I identify more performance improvements.

.

MartinKavik commented 4 years ago

What takes the most of time during the render? view function logic? Node<Msg> allocations? Or VDOM diff? Or DOM patching? Or something else / mix?

I think Elm optimizations are heavily based on immutability and purity - it doesn't need to call a function if the arguments are the same and the output is cached. And there won't be memory problems with those cached items thanks to immutability - it doesn't need to clone the whole item, it can only remember diffs. So I can imagine it's able to optimize also view function call because it just doesn't have to call unchanged branches/"sub-views" at all, especially in the combination with Html.Lazy. It's probably another reason why it's not recommended in Elm to store functions in your Model (functions can't be 100% compared, see this topic). And VDOM also use some JS-specific optimizations and other optimizations: https://elm-lang.org/news/blazing-fast-html-round-two. One of the optimizations is to reduce allocations as much as possible. It's a good idea also in Rust - it's well described in dodrio's blog post - read especially the section "Internal Design".

Your performance improvements are very impressive, however the code (especially under the title "The class selection") looks quite scary/cumbersome on the first glance. We will need to somehow polish/hide those performance optimizations "leaks". But I understand it's basically more a PoC / exploration than a final API, so you can ignore this comment šŸ™‚

@rebo Perhaps interesting discussion for you about style performance - https://github.com/elm/virtual-dom/issues/112

MartinKavik commented 4 years ago

Once you want to optimize benchmark create many rows (and probably also create rows or append rows) - look at my old investigation: https://github.com/initcrash/seed-quickstart/pull/1

rebo commented 4 years ago

What takes the most of time during the render? view function logic? Node<Msg> allocations? Or VDOM diff? Or DOM patching? Or something else / mix?

I think it is a bit of a mix and I will measure to be precise however I think the order of costs are as follows in a reasonably large view (1000+ nodes) (biggest first).

1) View function logic (inc Node allocations) 2) Dom patching 4) VDom diff

Although I need to measure this to be sure.

I think Elm optimizations are heavily based on immutability and purity - it doesn't need to call a function if the arguments are the same and the output is cached. And there won't be memory problems with those cached items thanks to immutability - it doesn't need to clone the whole item, it can only remember diffs. So I can imagine it's able to optimize also view function call because it just doesn't have to call unchanged branches/"sub-views" at all, especially in the combination with Html.Lazy.

Interesting, it must be something to do with this. It would explain why some partial update tests are so performant. I wil lreview the links.

And VDOM also use some JS-specific optimizations and other optimizations: https://elm-lang.org/news/blazing-fast-html-round-two. One of the optimizations is to reduce allocations as much as possible. It's a good idea also in Rust - it's well described in dodrio's blog post - read especially the section "Internal Design".

Will review thanks.

Your performance improvements are very impressive, however the code (especially under the title "The class selection") looks quite scary/cumbersome on the first glance. We will need to somehow polish/hide those performance optimizations "leaks". But I understand it's basically more a PoC / exploration than a final API, so you can ignore this comment šŸ™‚

Agreed, although much of the complexity arises from storing the selected state on the Row struct as opposed to a single model field. I think we could streamline this whole thing in a sensible way.

MartinKavik commented 4 years ago

Experiment inspired by Html.Lazy - https://gist.github.com/MartinKavik/678ad94e8a71dcea439c10a9f63387d9. All view function arguments are read as bytes and hashed. Function bodies are executed only when the arguments haven't been hashed yet or their hash changed. It would eliminate the need to call orders.skip and should theoretically improve performance. However I don't know how fast is hashing and there are probably some hidden traps in byte reading.

arn-the-long-beard commented 4 years ago

Hi guys ! Thank you so much @rebo for this issue. It helps a lot to understand the challenges we are facing and it does explain a lot about how web frameworks work :heartpulse:

I think Elm optimizations are heavily based on immutability and purity - it doesn't need to call a function if the arguments are the same and the output is cached.

Is the same as memoization ? Maybe we can do the same in Seed. I used memoization a lot in state-management.

Since I come from Angular, I can only say that Shadow DOM ( I thought it was Virtual Dom but no, damn it I am rusty :flushed: ) is used by Angular and that the benchmarks are pretty good at least on Angular 8 with the regular renderer. It was closed to vanilla javascript.

Now I never had the need to play with the renderer much so I do not know the Shadow DOM well.

But I know since Angular 8, we have access to a new renderer named Ivy.

From some post I found, it seems that Ivy is using an Incremental DOM.

https://blog.nrwl.io/understanding-angular-ivy-incremental-dom-and-virtual-dom-243be844bf36

I am not sure this concept can help us and if we can make it without breaking so much our code. Please tell me if I can help you in any way !

MartinKavik commented 4 years ago

Since I come from Angular, I can only say that Shadow DOM ( I thought it was Virtual Dom but no, damn it I am rusty šŸ˜³ ) is used by Angular

I would assume that frameworks like Angular use Shadow DOM to isolate components - e.g. your button component lives in a Shadow DOM. You get scoped styles for free for the button and a browser is happier because it doesn't have to update the entire DOM if you've changed only a small thing inside the Shadow DOM boundaries (e.g. you've only changed the button label). (Please somebody correct me if I'm wrong, it's only my guess.)

I think it would be hard to integrate Shadow DOM as a component wrapper to Seed because, well, Elm architecture doesn't have components. It makes more sense in the combination with Seed Hooks/Style, however it would complicate render algorithms a lot because we would need to communicate with another "layer" of JS Web API. And Rust/JS calls are generally slow + I assume we would discover many hidden traps without a reasonable increase in performance. Also we probably wouldn't be able to use "global" styles / CSS frameworks without problems.


Now I never had the need to play with the renderer much so I do not know the Shadow DOM well.

There are some examples of custom elements in the Seed repo - https://github.com/seed-rs/seed/tree/master/examples/custom_elements/src. They use Shadow DOM to "hide" their internal content from Seed VDOM, so they can modify their DOM as they wish.

So I assume that, for instance, https://www.fast.design/ would be easy to integrate to Seed apps because it should contain only Web Components / custom elements with Shadow DOM and all of them are described in a JSON files. So it would be possible to generate Rust types for them and create a component library even without Seed Hooks.


From some post I found, it seems that Ivy is using an Incremental DOM. https://blog.nrwl.io/understanding-angular-ivy-incremental-dom-and-virtual-dom-243be844bf36

Quote from the article: "Every component gets compiled into a series of instructions. These instructions create DOM trees and update them in-place when the data changes.".

If I'm not wrong it's how frameworks like Aurelia, Svelte or Imba work.

There are some obstacles to make this work in Seed:

In Seed, the most similar concept is probably @rebo's Hooks and Atoms.

Yeah, the rabbit hole is pretty deep, I'm glad that @rebo is exploring those concepts and optimizations because it's basically a full-time job for one person.

arn-the-long-beard commented 4 years ago

Thank you so much @MartinKavik for your explanations.

I would assume that frameworks like Angular use Shadow DOM to isolate components - e.g. your button component lives in a Shadow DOM. You get scoped styles for free for the button and a browser is happier because it doesn't have to update the entire DOM if you've changed only a small thing inside the Shadow DOM boundaries (e.g. you've only changed the button label). (Please somebody correct me if I'm wrong, it's only my guess.)

Yes it works this way :smiley: . That is why when your write a class in a angular component .scss or .css has a generated name at runtime also.

Yeah, the rabbit hole is pretty deep, I'm glad that @rebo is exploring those concepts and optimizations because it's basically a full-time job for one person.

I am lacking the full knowledge but I like also to make stuff faster. So I want to contribute helping in this side.

flosse commented 3 years ago

FYI: I just updated the js-framework-benchmark example to seed v0.8.0: https://github.com/krausest/js-framework-benchmark/pull/854

zzau13 commented 3 years ago

image

DeLorean is the same as the Seed runtime. But the implementation in wasm-bindgen far exceeds that of Seed, you can still get an idea. I hope help you.

evbo commented 2 years ago

Im confused because the "Benchmarketing" in the readme to this repo shows seed outperforming every other ui library.

Can someone please comment on what the "Benchmarketing" chart means relative to these benchmarks where Seed is not faster than most JS libraries?

zzau13 commented 2 years ago

https://krausest.github.io/js-framework-benchmark/2022/table_chrome_102.0.5005.61.html

flosse commented 2 years ago

Seed is not fast but it's also not slow :stuck_out_tongue_winking_eye:

It's probably more interesting what is important to your project. If performance really matters, take something else (and you still can use Rust/WASM for heavy background tasks). But if you want to profit from all the Rust benefits (compile time checks, maintainability, etc.) then you should consider a Rust frontend framework like seed :smile:

evbo commented 2 years ago

@flosse thank you, I guess as a newcomer it's a bit confusing. One naturally suspects Rust to be fast, they see a bar chart called "Benchmarketing" in your readme that illustrates Seed to be faster than any other JS library. Meanwhile the largest accepted benchmark test published by krausest says a different story.

So why not have the readme say this statement instead?:

Seed is not fast but it's also not slow

After researching this more, I realize why: There are no DOM bindings for WASM (yet). So until that happens, any DOM-related event (most everything in frontend UI) must pass through a middle layer like Javascript.

The above is dead wrong. Here's a proper explanation: https://github.com/WebAssembly/component-model/issues/31#issuecomment-1150231023

zzau13 commented 2 years ago

Delorean-rs uses wasm-bindgen giving the same results as vanilla javascript.

It is because they repaint unnecessary things and the tree difference algorithm does not exist or as if it did not exist.

evbo commented 2 years ago

@botika thanks. Yes what I posted above was misinformation I now realize.

I got clarification from wasm developers and sure enough, what you wrote is correct, native JS libraries are highly optimized. Rust libraries may take a while to catch up. Being a lower level language doesn't automatically mean you're fastest.