Simplify lit-html benchmark code

[aomarks]

A few simplifications to the lit-html benchmark implementations:

• Minor style updates (semicolons etc.).
• Update select and label properties in-place and remove guard directive, instead of creating new row objects every time.
• Remove some unnecessary attributes and use closest() to find the action and row id.
• Directly update row class on select instead of forcing an unnecessary render.

cc @justinfagnani @rictic @CaptainCodeman

[ryansolid]

My understanding is outside of the VanillaJS implementation storing the selected row is considered bad practice as if you were to take that state you would not be able to re-render from the data alone.

This was discussed at length here from this comment on. (Yeah I was guilty of doing this myself at first)

Just came up again in #519.

And is still under discussion in #430.

[CaptainCodeman]

Yeah, I think the idea is to demonstrate the library's ability to render and re-render as it would normally be used - adding vanilla JS to do updates means the library itself isn't really being benchmarked for those tests (and eventually, all framework implementations would do the same and produce near identical results).

The immutable updates + guards were used with that in mind and seemed to give the best performance assuming that constraint.

[justinfagnani]

Author of lit-html here.

I realize this is a very arguable point here, but as the designer of lit-html I have to say that this is actually idiomatic usage that the library is designed to allow for. At the very least it's not non-idomatic.

Because lit-html doesn't do a vdom-style diff over the whole template contents, it's actually quite safe to manually mutate DOM that's not dynamically controlled by lit-html. Given a template like:

html`
  <div id="static">Static stuff</div>
  <div id=${dynamicID}>${dynamicContent}</div>
`

It's guaranteed safe to: Modify all attributes and children of the first <div>. Modify any attributes but id on the second <div>. It's unsafe and not allowed to modify the id attribute or the children of the second <div>.

This has a number of good implications:

1. This is essentially how directives work. They receive direct access to DOM. (In fact you could just make a selection directive for this benchmark).
2. You can use browser editing like contenteditable on elements that don't have lit-html expressions in them.
3. lit-html will still function properly on subsequent renders and update the expressions correctly.

Another example of how lit-html is explicitly designed to allow this is that we support manually or third-party created DOM as values to bindings. This is so that mutations that are easier done imperatively are supported as a first-class technique, or so you can use another library seamlessly with lit-html. Again, this works because we don't do diffing.

let chart = makeD3BarChart(data);
render(html`
  <h1>Chart</h1>
  <div>${chart}</div>
`, document.body);

So in general the philosophy of lit-html is to be flexible and allow a wide-variety of ways of managing dome, from fully declarative and controlled by lit-html, to integrating with other libraries, to lit-html providing scaffolding for manual DOM updates.

In this case, if I were building a dynamic list with selection, I definitely would not re-render the list just to change one class name. After looking at the vanilla implementation and seeing that it directly updated the effect rows, of course I would suggest to do that with lit-html too because it's by far the more direct and efficient way to do it.

Also, somewhat independent of the direct manipulation conversation, keeping the selection state embedded with the data-model is pretty unrealistic as usually you'd have data from some existing source, and selection state managed by the UI.

As far as the effect on the results go, I think it's at least somewhat fair that the lit-html uses a technique that it's specifically designed to enable, especially if it's true that that technique doesn't work in other libraries because a re-render would wipe out the manual change. That it's safe and pretty easy to do something like this is a distinct positive feature.

I again realize this is a very arguable point, but I hope the above clears up the intent from the library design.

[justinfagnani]

@CaptainCodeman

The immutable updates + guards were used with that in mind and seemed to give the best performance assuming that constraint.

I don't think guard() is very idiomatic in this case, and doesn't seem to give better performance. guard() should be use pretty sparingly, once you've identified that a sub-template is a perf bottleneck. In this case the template being guarded is simple enough that the perf seems to be the same with or without, but guard basically requires the partial immutable data pattern that's there now. That's a bit of an odd pattern to me, but without knowing that it's there for guard() you might think that all the data is intended to be immutable, yet the list itself is mutated.

[ryansolid]

@justinfagnani

I can see why you'd have that perspective. In fact in #430 I suggested that LitHTML might have a similar perspective to myself or the Surplus author. That being said many frameworks are not VDOM, deal directly with the DOM yet still don't depend on storing the selected TR as state.

The argument that made me change my mind on the matter was what if your initial state had 1000 rows already and row id 10 selected. Would this implementation properly render that? I'm not saying not to mutate the DOM directly or after the fact (although there are some parties that firmly feel this doen't belong in userland) but that the data state for rendering should be based off the row id and not a stored TR. I don't know enough about LitHTML to weigh in on idiomatic but not being able to completely re-render any at rest state from scratch purely from the data seems to go against the spirit of the benchmark.

[CaptainCodeman]

Surely the point of this project is to provide a way to compare the performance of various frameworks and libraries to do roughly the same thing? Yes, bypassing them can be faster and a potential optimization when using any of them, but we want to see relative performance of each framework to do the rendering using that framework to mirror the more typical usage where I think it's reasonable to say that most people want to mutate a model and have the state reflected in the UI.

Maybe by 2020, all frameworks will just bootstrap some WASM ...

[leeoniya]

Because lit-html doesn't do a vdom-style diff over the whole template contents, it's actually quite safe to manually mutate DOM that's not dynamically controlled by lit-html.

@justinfagnani this is also true of many vdom libs that simply ignore dom nodes or attrs they did not create. domvm, for instance does this. it diffs old vdom against new vdom, so anything you do to real dom (that's not managed explicitly by vdom templates) becomes effectively transparent.

in fact, many of us have been removing these imperative vanillajs optimizations from our impls over time, such as explicit event delegation at the <tr> <table> and dom-climbing/class-testing to avoid binding 2 handlers per row or storing the row id in 2 places.

the idea of a framework is not to show off how it can avoid interfering with imperative dom manip for optimizations; it's to reduce the necessity for having to do them in the first place.

[localvoid]

I am curious why libraries that already has a huge advantage in this benchmark (small ratio of dynamic data bindings per dom element, zero tests that will check how library performs when application is decomposed into small reusable blocks) are trying so hard to avoid using their primitives.

When I saw how lit-html mutated state in remove row and swap rows tests (before this micro optimizations), I've changed ivi implementation to make sure that it mutates state with the same slow implementation. All this micro optimizations just show that some library authors have zero interest in gaining any valuable information from this benchmark.

[Freak613]

Merging this we can completely wipe out "selected" test case, as it'll start spreading and, surprisingly, ability to go to the DOM level will be found in every library.

To push that, author should answer himself the question "What we're measuring here?" Browser performance in switching classes? This is not a browser benchmark.

If you compiled native switching from developers code, good job, you figured out how to automatically lift and fold complex computations, and I definitely would like to check it out. If you wrapped it in directive/helper, something that shipped with the library - fine.

The case here is to demonstrate shine of the library, using it in idiomatic way, provided by lib authors. Correct me if I'm wrong, but I think that DOM insertion in lit was added to seamlessly integrate third-party code which outputs DOM nodes, and it's not a regular way of doing things. If it's idiomatically, to write native DOM code for each updating case, then the question is why do I need the library.

For lit, probably, it will be enough to create some directive for lifting computations and ship it with the library. However, I see that application of such directives/helpers are very limiting. Okay, you created directive for lifting class switching. What if I need to update text in one node? Another directive?

[ryansolid]

I know where some of those comments are directed... And maybe at my day job I just have the mundane task of dealing with a lot of grids and lists setting a class or attribute based on selection happens to be way more common (media/user management in education industry). But my team uses KnockoutJS in our legacy WebApp and number of times I have train or help junior devs with this is non-trivial. To the point handling selection was on my shortlist. It wasn't to game, it is seriously a pain. Mapping the selected signal over the model is natural with reasonable performance on update, but unlike top down has to be done upfront and synchronized so specialized and awkward. Similarly wrapping a computed check on every row doesn't scale (especially how poor Knockouts performance is) with large lists. So complete acknowledgement of the cost there. Even before I saw Surplus solution I was doing something similar. It's just such a common problem and 99% it's a class since it's visual as parent is handling the functional selected state anyway. It could also be just the use of Shadow DOM which means actual cascading CSS instead of Styling as JS, leads to less explicit multi-level style bindings, and :host selector theming.

As I said before in #430, I suspect libraries used in conjunction with Web Components are often likely to see less wrong with direct DOM manipulations because their Components are DOM elements. The event.target could be their Component. And while clearly they aren't putting the overhead of making each row a Custom Element, it doesn't raise the same immediate red flags, especially around event handling. What should raise flags though is recognizing when the rendered state isn't purely data driven. At any point of rest the DOM should be able to blown away and reconstructed from the data. I believe that is floor on what can be considered data driven.

[WebReflection]

Merging this we can completely wipe out "selected" test case, as it'll start spreading and, surprisingly, ability to go to the DOM level will be found in every library.

Yup, and I can guarantee that lighterhtml tests will follow up any shortcut taken in here, since you can modify directly nodes with it too.

@justinfagnani the whole point of these libraries is that they are cheap to update, no matter how much data they consume. Bypassing a render even once, would simply show how weak instead are these libraries, if manual/vanilla JS hacks are needed to score any better.

I understand it might hurt seeing lighterhtml slightly ahead in some case, but if we start bypassing our own libraries primitives, we should also declare without shame that we failed.

As of today, these two are the fastest VDOM-less micro libraries out there, and I don't see why should we try any harder to benchmark what these libraries users will never write in their daily real-world code.

As summary, if you want to go as close as possible, or ahead of, lighterhtml, I suggest you use also my .closest(...) approach, written actually to simply have cleaner code instead of that ugly parentNode.id || ... hacky dance, and avoid re-assigning objects in the keyed test (IIRC I've left that part in the non keyed since I don't care there about the object being the actual key).

[Freak613]

@ryansolid nothing personal, just my thoughts.

Unfortunately, I can't agree. Lit is a library of building virtual view model from templates (via TemplateResult-s) and applying this structure to the DOM. It has same relation to the DOM as any other VDOM library. And it doesn't have Components concept. However there is LitElement about WC but this is not the subject of this PR.

Lit's html doesn't produce DOM nodes, but there are other libraries built over the concept of rebuilding real DOM nodes on every turn. They have much more in common with the DOM, producing it, and they points out at potential problems of this approach (again, I do not criticize).

I know that @justinfagnani is on close work of standardization Lit concepts and approaches, but today it's not in the browsers nor we know in what form it will be. And that fact or the fact that it happens to be used to build WebComponents somewhere, doesn't make it closer to the DOM. It's just a rendering engine with significantly reduced overhead. WebComponents can be rendered even with React.

[ryansolid]

@Freak613 Don't get me wrong. I don't think it is necessarily a defensible(correct) position and I recognize mechanically these libraries are not that different than a VDOM library. Just that I suspect people who work around and with WebComponents in mind are more likely to have blurred the line without paying it much heed. Especially libraries being rendered inside Custom Elements. When your 'this' is a DOM element how abstracted do you feel? But I can leave that observation for myself as while I see correlation, as you said it has nothing necessarily to do with this PR.

[justinfagnani]

@leeoniya

the idea of a framework is not to show off how it can avoid interfering with imperative dom manip for optimizations; it's to reduce the necessity for having to do them in the first place.

That's not necessarily the idea behind lit-html. lit-html is there to offer easy-to-use DOM updates when it's the best option, and to try to make it the best option in the largest number of cases. Philosophically we have no intention of monopolizing the DOM even within the same context. Also, lit-html isn't a framework, so may that's where the philosophical difference is from.

@ryansolid

The argument that made me change my mind on the matter was what if your initial state had 1000 rows already and row id 10 selected. Would this implementation properly render that? I'm not saying not to mutate the DOM directly or after the fact (although there are some parties that firmly feel this doen't belong in userland) but that the data state for rendering should be based off the row id and not a stored TR.

I think this is a fair critique - given initial data with a selection (I still prefer the selection state separate from the data model) this should render correctly. This would also align with setting the class declaratively. I don't think storing a DOM reference is necessarily an anti-pattern - It's how many directives work, and directives are intended as a user extension point - but that should be a transparent optimization.

We have rough ideas for mutation hints that would perform essentially the same optimization here, but on a public-API path. I think we could roll-back the direct manipulation and revisit if and once we have that.

[localvoid]

Here is how easy it will be to abuse the same technique in React (short-circuit diffing and use direct DOM manipulations): https://codesandbox.io/s/ojrp4pr54y . I can even create it as a reusable component. Everyone who is using such techniques just shows how weak their rendering engine.

[ryansolid]

@localvoid Not gonna lie I think the possibility of something like that being done in React is great. Directives for React. Hooks in general allowing Top Down renderers to mimic Fine Grained in my opinion the greatest thing the React team has ever done (including inventing React). I believe the application of this technique is profound outside of React itself. If done well enough why would you ever do this a different way? At what point is this using the tools given the best way possible? Who cares how 'weak' something is if it is no longer applicable.

[localvoid]

@ryansolid If done well enough why would you ever do this a different way?

Because the only reason why you doing it is to get better results in this benchmark. Here is your TodoMVC implementation: https://github.com/ryansolid/solid-todomvc/blob/c230edc43d1a1207d94770fb414a5faf955efc73/src/index.jsx#L96

[ryansolid]

Todos carry the done state with them back to the database where in general the selected state here is presentation state. In the Todos case it doesn't make sense to make this optimization since it is part of the data model. The more common scenario, which I believe this benchmark represents, is you are just temporarily selecting something for performing some sort of operation on it. It doesn't belong on the model which could be shared outside of this specific context.

Now since this Benchmark doesn't do anything with the selected state and leaves it open to set up the model to fit your libraries change detection mechanism, its arguable that the scenario here could be the Todos one. But my impression plus a couple recent quotes make me think it isn't:

@justinfagnani

(I still prefer the selection state separate from the data model)

@leeoniya

this is a thorny one. in my view, shoving "selected" into each item is kinda polluting the models/data with presentation-layer state.

[localvoid]

I am talking about editing state:

1. https://github.com/ryansolid/solid-todomvc/blob/c230edc43d1a1207d94770fb414a5faf955efc73/src/index.jsx#L96
2. https://github.com/ryansolid/solid-todomvc/blob/c230edc43d1a1207d94770fb414a5faf955efc73/src/index.jsx#L107

[ryansolid]

You are absolutely correct. Sorry, I feel sort of dumb now that given the thousand times I've looked at TodoMVC I never made the association it was the same thing. Even though its the single select I use here it's the multi-select that I use in the majority cases elsewhere. On the positive this means the single select directive might be more applicable than I was originally thinking. I'm going to explore that more. Thank you.

[krausest]

So what are we going to do? I see three options:

1. We can leave the benchmark as is if we (and this means especially @justinfagnani and @aomarks) agree that (data driven) implementations should not use dom access simply because we gain no insights from doing that; we're already measuring the performance of vanilla-js implementations. We'd use the (well working) community review process that identifies overly clever optimisations.

2. I could add a rule to the benchmark that forbids dom access for data driven frameworks (but formulating that rule might turn out to be hard) and close such PRs. Maybe this would mean we'd lose some competitors (or their support). Not my favourite option but maybe necessary.

3. We could distinguish between implementations that use dom manipulation and those that don't. The results front end would not allow direct comparisons. IMO not a good addition to the benchmark and a mostly useless comparison for implementations that use dom manipulation.

[justinfagnani]

I'm very ok with removing the direct DOM manipulation here for now.

The requirement that initial rendering be able to render selection state is a good one, and I think that should be encoded as a rule, and maybe a test (ie, verify that one a certain row is selected in one of the creation tests).

For lit-html specifically, since we had already been considering ways to let code trigger re-rendering a portion of a template because this use-case is pretty common, we'll continue to think and work on that and revisit this here if we have a 1st class, documented, and still within the template declaration way of doing this.

How's that sound?

[ryansolid]

Yeah I think libraries are actively looking for ways to do finer grain changes declaratively even if the current solutions perhaps aren't ideal. The baseline being the implementation able to render from an initial dataset is something I believe everyone can agree on. I'm not sure how to enforce it since each implementation is responsible for building out it's data, but I believe it's one of the only thing that no one has opposed across all these threads including #430 .

[aomarks]

SGTM too. Big +1 to finding a way to encode the requirement discussed here as a new benchmark/test, since I think that would naturally push implementations in the direction we want without hypotheticals. I'd like to think the ideal set of requirements for any implementation here could just be 1) "You perform all of the benchmarks correctly" and 2) "You use only the idiomatic style of your library".

Happy to close this PR, but I do think the first 3/4 commits could still be committed, because I don't think they touch on the objections raised here. But I can send another PR with just those commits to keep things clear.

[krausest]

@aomarks @justinfagnani Thanks!

@aomarks Closing and submitting a new PR would be perfect.

Please take a look at #522 to vote for a new rule or submit a well formulated one.

[adamhaile]

a huge advantage in this benchmark

This has been repeated several times and deserves to be rebutted.

1. This benchmark is heavily biased towards creation and teardown which is famously an area where vdom-like approaches have an advantage over fine-grained ones. Fine-grained approaches have to generate bookkeeping to track which data is tied to which outputs, which pays off once you start seeing changes but which is pure cost during creation and teardown.

2. Bindings-per-node is a red herring, since fine-grained approaches let you choose just how fine-grained you want to be. Just as an example Surplus generates code that makes a single binding (computation) per node, so its cost is exactly equivalent to VDOM approaches. VDOM libs make a property object per node, Surplus makes a closure per node, both of which have equivalent cost ... until you start seeing changes, at which point Surplus has zero memory cost, but the VDOM libs are still generating lots of property objects.

3. The benchmark is also biased towards VDOM-like approaches in that there are no tests whose work size scales with the change size rather than the data size. Updating 1/10th of the nodes still scales with data size, and as has been much discussed, the selected test requires scanning all nodes, so again it scales with data size. An example of a case where fine-grained approaches shine is drag-and-drop on a fairly complex scene.

4. Finally, VDOM approaches have a major and entirely unrealistic advantage in this test in that with such a simple app, the diff algorithm at the heart of VDOM libs stays merely polymorphic, whereas in a real app, it would quickly go megamorphic and get de-optimized by the js VM. The other points above are cases where this benchmark favors VDOM libs, but this last is much worse, in that: it produces results which are unrealistic and misleading.

All that said, this is still the best and most useful benchmark out there, and a huge thanks to @krausest for his work on it.

[localvoid]

1. This benchmark is heavily biased towards creation and teardown which is famously an area where vdom-like approaches have an advantage over fine-grained ones.

When ratio of data bindings per DOM element is so low, vdom generates alot of vdom nodes just to render it once, and then they become useless. Libraries like lit-html clone large DOM chunks and generate small graph with dynamic parts. Libraries with optimized KVO bindings also generate a way much smaller graph than vdom libraries in this benchmark.

2. Bindings-per-node is a red herring, since fine-grained approaches let you choose just how fine-grained you want to be. Just as an example Surplus generates code that makes a single binding (computation) per node, so its cost is exactly equivalent to VDOM approaches.

If it is so efficient, why you need to create workarounds and reduce bindings per node even lower, especially when such optimizations can be easily implemented for all other libraries.

3. The benchmark is also biased towards VDOM-like approaches in that there are no tests whose work size scales with the change size rather than the data size. Updating 1/10th of the nodes still scales with data size

For libraries with fine-grained bindings it scales with multiplier 1 and for most vdom implementations with multiplier > 7.

4. Finally, VDOM approaches have a major and entirely unrealistic advantage in this test in that with such a simple app, the diff algorithm at the heart of VDOM libs stays merely polymorphic, whereas in a real app, it would quickly go megamorphic and get de-optimized by the js VM.

Proofs? I can easily add code that will pollute stub cache with thousands megamorphic call-sites, render alot of different elements with different attributes, components, etc to make sure that JIT collects information about all possible types and there won't be any noticeable changes in performance. In fact, ivi is using code that should perform worse in micro benchmarks like this, but will perform better when stub cache is heavily polluted and there are alot of collisions.

[leeoniya]

1. This benchmark is heavily biased towards creation and teardown which is famously an area where vdom-like approaches have an advantage over fine-grained ones.

yeah, wat? vdom libs create a parallel dom tree in addition to binding the data, in addition to creating the actual dom. i'm definitely missing how it is biased towards them 😕

[ryansolid]

1 & 2

The cost is generally greater on creation/teardown for fine grained. @adamhaile was just saying that more bindings doesn't necessarily mean more computations. But those computations are generally more expensive create and destroy. Disposal always does work to account for the classic lapsed listener problem of the observer pattern. While a VDOM might produce a bunch of useless nodes the overall cost of creating the tree is cheaper. Is it even cheaper for litHTML? Definitely, even further exemplified by DOMC and Stage0.

3

@adamhaile didn't say that the partial update wasn't a good test for fine grained, just that it doesn't exemplify the advantage that comes from this approach as the data set size is still a factor. It is the definitely a better test for fine grained in this benchmark.

4 Honestly I will leave that up to you guys to figure out.

I know saying the benchmark is biased towards VDOM is inflammatory but I can relate with the frustration of the same misinformation being pedaled. I know it goes both ways especially with the confusion with lifecycle methods(is event sub/unsub really lifecycle methods? under a certain light). Generally I've been ignoring it but I'm glad Adam spoke out.

I wouldn't say this benchmark in particular in comparison to pretty much every popular benchmark out there is biased that way. However, it is most definitely hasn't been biased towards fine grained. This tired argument about number of bindings per DOM node has no place. The benchmark has twice the number tests around creation and tear down than smaller updates and those smaller updates are still constrained by list size. A benchmark biased for fine grain would not look like this and most definitely would not have multiple tests for creating nodes en masse.

[localvoid]

@adamhaile was just saying that more bindings doesn't necessarily mean more computations.

Are you saying that there will be no difference between <a>{row.label()}</a> and <a>{row.label}</a>, or more complicated cases with dynamic bindings and conditional rendering, etc? For vdom there is no difference, everything is treated like it is a dynamic binding. Does it mean that vdom will be faster than fine-grained solutions when the ratio of dynamic bindings is high? No. But it is definitely should be slower when the ratio is low.

UPDATE: Didn't understood what you initially meant by "more computations". But if you saying that "those computations are generally more expensive create and destroy.", does it mean that if tests had higher ratio of bindings, then this libraries will be slower in test cases with "creation and teardown" ? So they obviously should benefit from low ratios. I am confused, 2 dynamic bindings per 8 DOM elements makes it biased towards VDOM libraries in this test cases so that it requires to use workarounds and reduce it to 1 dynamic binding to make it fair?

[ryansolid]

I meant that every binding on a node in Surplus gets wrapped in the same computed.(Solid doesn't work that way currently I do one computation per binding but I'm evaluating changing that). There is an unavoidable cost for that first one. And children are wrapped in a separate one than their parents.

So if you had

<div style={style()} class={class()} name={dynamic()} />

You'd get something like:

const el = document.createElement('div');
S(() => {
  Object.assign(el.style, style());
  el.className = class();
  el.name = dynamic();
})

I believe it's just per node but realistically there is nothing to stop all bindings among siblings in non-dynamic situations to use the same computation.

Fine Grain won't always avoid making more computations even if it doesn't necessarily have to from just the fact of there being more bindings. VDOM is always the same. The question is how expensive each computation is by comparison? 2 computations per row here would set Surplus considerably back. Generally it's pretty common for a fine grained library to have a couple computations in similar situations. It probably won't scale out any worse than that but its enough that VDOM libraries would generally be faster on create/teardown. That clearly isn't happening in the implementation for this benchmark as the numbers show although I don't believe that was always the case. Back when Chrome was slower I remember ivi being faster in this particular area for a long time. Maybe that's a testament to ivi rather than VDOM in general. It is fine grain's weakness which is why it gets so much focus.

[adamhaile]

Libraries with optimized KVO bindings also generate a way much smaller graph than vdom libraries in this benchmark.

That's not because they're KVO, it's because they use a compiler. VDOM libs could do the same. I guess rawact is exploring that.

(S is neither KV nor O, BTW, but the difference doesn't matter here. KV implies that there's an object with keyed properties, while in S signals are first-class entities that each hold a single reactive value. O means simple observer-pattern depth-first change propagation. That's faster but harder to reason about than what S does. S has a scheduler to make sure changes propagate atomically and glitch free.)

If it is so efficient, why you need to create workarounds and reduce bindings per node even lower, especially when such optimizations can be easily implemented for all other libraries.

"Easily" like the React example you posted, where it took how many lines, how many hooks and forwardRefs and ceremony, to do what takes 2 lines in Surplus? That's not an accident. Surplus is designed to make such things easy. Not because modifying DOM nodes all over the place is a good idea (it's not), but because S is built around the idea (coming from SRP) of declarative effects. So modifying DOM in an S computation can be both idiomatic and predictable.

Saying that S shouldn't because other libraries made choices that make it hard is demanding that everybody else adopt the cons of your approach.

For libraries with fine-grained bindings it scales with multiplier 1 and for most vdom implementations with multiplier > 7.

1 and 7 whats? Sorry, can't read your mind here. In either case, it sounds like you're talking constant factors, while I was speaking about big O.

Proofs?

Fair question. I've definitely seen it be a factor in React microbenchmarks. I haven't dumped optimization traces from this benchmark to prove it, but I don't see anything in the approach or data size that make me think it wouldn't.

[adamhaile]

1. This benchmark is heavily biased towards creation and teardown which is famously an area where vdom-like approaches have an advantage over fine-grained ones.

yeah, wat? vdom libs create a parallel dom tree in addition to binding the data, in addition to creating the actual dom. i'm definitely missing how it is biased towards them 😕

That brings it closer but still doesn't close the gap, for two reasons:

1. At base, you're creating a single-linked tree, while fine-grained reactive libs are creating a double-linked graph. That's a necessity of the approach: sources need to know what consumers to invalidate when they change, and consumers need to know what sources to deregister from when they leave the graph. The double-linking and the fact that it's a graph not a tree make construction more expensive.

2. I said creation and teardown. You just de-ref the root of your vnode tree and you're done.[1] Since the dependency graph is double linked, I need to take the graph apart registration by registration. O(1) vs O(N).

But hey, part of the point of S is to show that the famous "bookkeeping" cost of fine-grained reactivity is overhyped[2], So if you don't think it's there ... win? 😁

1 - That reminds me, way number 5 that this benchmark is biased towards VDOM libs: VDOM libs create tons of short-lived objects as though GC cost didn't exist. This benchmark makes no attempt to account for GC cost -- GC usually runs after the timer stops. I don't have a good solution for this - every time I've tried to quantify GC cost in Chrome it has introduced too much noise to be useful. My current approach is to measure heap size before and after and assume that cost scales with memory allocation. Maybe chromedriver has more analytics to offer.

2 - See for example Dan Abramov's comment in React as a UI Runtime about the cost of "fine-grained listeners."

[localvoid]

VDOM libs could do the same. I guess rawact is exploring that.

It is not so easy when you think about edge cases and performance on the other side of the spectrum when everything is decomposed into small reusable blocks, not like this benchmark. In many react applications there is more component instances than DOM elements.

I know at least 2 other developers with a good understanding of this problem space that tried to solve it, and there are still no solutions. At some point I even gave up on vdom and tried to explore templates + incremental dom, but I couldn't find any good solutions to some edge cases.

"Easily" like the React example you posted, where it took how many lines, how many hooks and forwardRefs and ceremony, to do what takes 2 lines in Surplus?

const ref = useRef(null);
useLayoutEffect(() => { ref.current.className = "abc"; }, []);
return <div ref={ref}></div>;

forwardRefs are irrelevant, they are solving React specific problems with their APIs.

Saying that S shouldn't because other libraries made choices that make it hard is demanding that everybody else adopt the cons of your approach.

I really don't understand why you think that it is hard. Maybe you think that it is hard because you don't work with React. When I look at your implementation, it isn't so obvious to me and I can start feeling that it is harder.

1 and 7 whats? Sorry, can't read your mind here. In either case, it sounds like you're talking constant factors, while I was speaking about big O.

I understand that you are speaking about big O, but software engineering isn't a computer science, it is important to take into account input data and constant factors.

Test case with DnD and complex scene is quite easy to optimize in vdom libraries, so there won't be any difference in results because paint/reflow will be the biggest factor. But this partial update test case that triggers diffing of > 800 vdom nodes isn't so easy to optimize unless you want to lose performance in create/replace/remove tests, so it can actually showcase the biggest drawback of vdom libraries.

I've definitely seen it be a factor in React microbenchmarks. I haven't dumped optimization traces from this benchmark to prove it, but I don't see anything in the approach or data size that make me think it wouldn't.

I am pretty sure that ~90% of libraries in this benchmark aren't optimized for this, it is not about vdom/non-vdom libraries.

I said creation and teardown. You just de-ref the root of your vnode tree and you're done.[1] Since the dependency graph is double linked, I need to take the graph apart registration by registration. O(1) vs O(N).

And who will invoke componentDidUnmount(), etc? In your O(N), N is a number of bindings, with vdom N is a number of vnodes. It is just way much cheaper to go through vdom and invoke this callbacks and cleanup.

VDOM libs create tons of short-lived objects as though GC cost didn't exist.

1. Single-phase diffing algorithms doesn't create tons of short-lived objects.
2. Because there is a huge number of DOM elements in this benchmark, some libraries are actually triggering GC during append rows to large table, and some libraries are probably triggering it in other tests.
3. It would be better if we start using some examples with absolute numbers, because when everyone is talking shit about vdom memory overhead with their 100x improvement, it is often means that they reduced memory overhead from 1kb to 10b.