The weight is killing innovations in `JS` frameworks

[ClassicOldSong]

In short, the weight put JavaScript frameworks in an unfair competition.

If this benchmark is called "web-framework-benchmark", I would have no objection on this weight, but... it's called "js-framework-benchmark" so...

The newly added weight encourage frameworks change into "compile time optimizations" that work only in browsers - frameworks who use innerHTML and cloneNode to gain speed in node creation, but only in browsers, while originally JS frameworks can have a much wider availability in multiple other scenarios such as iOS and Android app development(e.g. with NativeScript with DOMiNATIVE), and even embedded GUI development(what I'm working on right now), but these environments are very unlikely to provide a built-in XML parser and even if, won't benefit much from doing so. While document.createElement can be provided and translated very well in multiple GUI implementations, using the same API also provides an even ground for each frameworks to compete about their "JS" part - innerHTML and cloneNode are provided by browsers, not JS itself.

Also, in my opinion, node creation speed isn't that important that needs a ~60% weight, but content updating speed affects user experience in a bigger aspect.

ref: #1341 #1233

[ClassicOldSong]

I must point the fact out that, JavaScript is not a BROWSER ONLY language, and it can have much wider usages other than browsers and NodeJS. Since JavaScript frameworks have the most versatile experience and accumulation in the GUI domain, we can make them being useful to other fields other than browsers.

But this needs us to think in an open mind. People who didn't realize that JS can do more than browsers, we need to let them know.

[leeoniya]

If this benchmark is called "web-framework-benchmark", I would have no objection on this weight, but... it's called "js-framework-benchmark" so...

yes, effectively this is a "web-framework-benchmark". or if you want to be even more pedantic, it's a "dom-framework-benchmark", since Three.js can also be considered a "web" framework, as can any gui library compiled to WASM + WebGL. some people consider Express.js a js framework :man_shrugging:.

nevertheless, "js framework" has a de-facto, established meaning of "dom framework" [1], much like "Coke" refers to a drink that no longer contains cocaine. most people recognize this repo for what it is, i think. im not sure if getting hung up on semantics is a worthwhile endeavor.

[1] https://www.google.com/search?q=js+framework

[ClassicOldSong]

Actually DOM framework still isn't accurate - innerHTML isn't strictly a DOM thing

Anyway this weight still encourages frameworks to "optimize" only for browsers and eventually will break their ability to run elsewhere, narrowing their actual usability.

I want to raise awareness of the fact that JS frameworks appear here can have greater potentials, and I don't want the weight to kill them.

[ClassicOldSong]

There's already an example that broke: Svelte 4 runs flawlessly in NativeScript and my embedded environment, but Svelte 5 made the innerHTML optimized compiler the only option, rendering it completely unusable in these scenarios.

Solid can be hacked to still use createElement and it's now functioning well in NativeScript, they even did a livestream about this days ago.

Please don't narrow possibilities down.

[ClassicOldSong]

You want to keep the name"js-framework-benchmark" is fine, but could we make the other potential about those frameworks visible and noticeable?

[leeoniya]

innerHTML isn't strictly a DOM thing

it quite literally is: https://w3c.github.io/DOM-Parsing/#dom-innerhtml-innerhtml, and has nothing to do with ECMAScript (JS). what you probably mean is that some JS frameworks adopted this well-known DOM getter/setter name for their own concept of "string of markup/html". if a framework's strategy of pre-compilation is not compatible with a non-standard DOM environment, then this sounds like something you should discuss with either the framework authors, or the authors of the environment that is DOMish-but-not-really.

but Svelte 5 made the innerHTML optimized compiler the only option, rendering it completely unusable in these scenarios.

and if that is faster in the standard DOM environment, then it was probably the correct choice for them, given that's likely the overwhelming majority of Svelte's (Vercel's) target audience.

but could we make the other potential about those frameworks visible and noticeable?

React has the ability to target non-dom environments (via React Native), and it is plenty visible/competitive here, right?

[ClassicOldSong]

I mean, over 90% frameworks here can work in non-browser environments, they just didn't realize that.

DOM wasn't built only for HTML in the beginning, what you're referring to is HTML DOM, and innerHTML clearly isn't a DOM level-one api.

RN handles non-dom environment itself, but what I'm talking about is making most frameworks have this ability without creating wheels again and again themselves. This is totally different than RN's approach.

We could just implement a minimal DOM in any new GUI environment and boom we have lots of available frameworks to run already, but if we need to talk to every framework author about writing a specific implementation again? That's not ideal.

[ClassicOldSong]

What they could clearly realize is, their score were affected even more with creation speed and they'll tend to use the more browser specific APIs, without the knowledge of they can have more potentials.

This is pretty bad since it's clearly harmful to diversity.

[leeoniya]

I mean, over 90% frameworks here can work in non-browser environments,

i don't really understand what you're trying to say here. with a proper environment shim everything can work everywhere. Wine/Proton exists and all the software built for Windows "just never realized" that it can also run on Linux?...because of Wine?

all of these frameworks can obviously run in non-dom environments as long as those environments have the features they rely on from the DOM. and that is a pretty huge statement to make because this benchmarks measures the cost of doing proper css and html compliant layout, styling, event model, text layout. any shimmed environment that does not provide all of this cannot be compared (in this benchmark) to one that does.

[ClassicOldSong]

Idk if you have worked with any embedded environments, but their resources are very limited and mostly will not benefit from having a XML parser built with their runtime.

I've clearly stated this in the beginning.

Shims are fine, until they're not. Like Wine more or less still affects performance, that's why we still need native Linux executables.

[ClassicOldSong]

Further more, CSS has nothing to do with these frameworks. They're interfacing DOM only, while browsers handle the CSS property bindings from their DOM.

Providing a minimal enough shim to run these frameworks is not huge and I've already done it on a device that has only 8MB of RAM, with XIP it could be even 2MB. CSS isn't needed since it's not the only styling method in the world.

[leeoniya]

have you asked the Svelte authors (or authors of any fast frameworks here) if they would consider reducing their DOM performance and featureset to maintain compatibility with an environment that has a reduced subset of the DOM and no CSS layout? for frameworks that target the browser as their first-class/priority, i'm guessing the answer will universally be "no".

Further more, CSS has nothing to do with these frameworks.

but it has something to do with this benchmark, because it measures the time required to do proper CSS parsing and layout of the rendered content. if your framework or environment does not render with the same rules as the DOM/CSS, then it is not going to be measured against the rest that do.

This is pretty bad since it's clearly harmful to diversity.

and Formula 1 cars dont race in NASCAR races. all participants in the benchmark must play by the same rules. in this benchmark, they all must render using the same set of modern browser tech: DOM, CSS, HTML, JS, WASM. they cannot render on Canvas or use WebGL.

[ClassicOldSong]

have you asked the Svelte authors (or authors of any fast frameworks here) if they would consider reducing their DOM performance and featureset to maintain compatibility with an environment that has a reduced subset of the DOM and no CSS layout?

Yes I have and the answer is "yes" by considering adding back the createElement way, though time isn't guaranteed.

What I'm arguing about is don't lose potentials, while providing compile options that do both is even better than having only one choice.

[ClassicOldSong]

You still didn't get the biggest issue here, which is people don't know they have more potentials and they will lose them when the benchmark doesn't tell them and encourage them to optimize for only one direction.

The problem here is hiding choices.

[localvoid]

providing compile options that do both is even better than having only one choice.

Why should I choose a subset of an awful DOM API as a universal retained mode UI API?

innerHTML/cloneNode optimization isn't the only thing that is used to optimize specifically for a browser environment. For example, in the ivi library, Right-To-Left update algorithm or children list diffing that tries to find minimal DOM ops won't be useful outside of a browser environment.

[ClassicOldSong]

Why should I choose a subset of an awful DOM API as a universal retained mode UI API?

Certainly you can choose not to, but you'll lose the ability to run on more platforms.

And, I think you haven't read carefully enough: we're talking about running outside of browsers that can't provide ultimately the same APIs due to multiple reasons, and your quote just said it's even better if you can choose when you need those fancy optimizations, and your code can still work when these optimizations are not available.

[ClassicOldSong]

Right-To-Left update algorithm or children list diffing that tries to find minimal DOM ops won't be useful outside of a browser environment.

Actually they're still usable since it's using only DOM level one APIs, that is available in ANY DOM implementation.

Go check undom-ng, which is the actual DOM implementation powering DOMiNATIVE and you'll understand what I mean.

[localvoid]

Actually they're still usable

Usable doesn't mean that they will be useful. If I am going to target different platform, I am going to use different architecture/algorithms/etc and I am sure as hell I don't want to target an awful DOM API when I am not constrained by a browser environment, especially if I am going to optimize for a resource constrained embedded environment.

[ClassicOldSong]

I wrote my own frameworks, and I have a new framework that does DOM diffing also(but with my own algorithm), working flawlessly in my DOM impl and in browser, and Solid/Vue/React all ran smoothly in my DOM implementation. DOM itself isn't a browser specific thing but HTML DOM is. When DOM points to generic XML documents, innerHTML isn't appropriate anymore. And as it's a fact that it's a MODEL, it can actually run without any XML like backend.

I believe ivi might run well on NativeScript writing native iOS and Android apps right now if it's not using the innerHTML trick.

It's up to you frameworks authors to determine if you want to embrace more platforms, or limiting your framework to browser only.

[ClassicOldSong]

I am going to use different architecture/algorithms/etc

Sure thing, it's your choice to build wheels again and again for your own interests, but I am trying to let people taking benefits from their existing work without having to do duplicated labor, and we can maximize the experience and accumulations frontend frameworks have already gained by reusing the existing code and libraries.

And just lacking innerHTML means "awful" is really entertaining to be heard.

[localvoid]

And just lacking innerHTML means "awful" is really entertaining to be heard.

I've never said that "lacking innerHTML means awful" or anything like that.

[ClassicOldSong]

I've never said that "lacking innerHTML means awful" or anything like that.

Why should I choose a subset of an awful DOM API as a universal retained mode UI API?

I am sure as hell I don't want to target an awful DOM API

Then could you clarify what "awful" are you referring to?

[syduki]

Also, in my opinion, node creation speed isn't that important that needs a ~60% weight, but content updating speed affects user experience in a bigger aspect.

While I am agree with you on that, I must admit that the discussion is going in the wrong way and I think you oversight the whole essence of this benchmark.

You are asking for universality and diversity in a performance focused environment where everything is about specificity and narrowness. You are asking about standards while this benchmark is encouraging the tricks.

This benchmark, as any other, is not a specification for framework implementers, nor a howto for framework users, as you may note, the are only two weak rules, both for the benchmark implementation, i.e. an advice to write it in an idiomatic way and to be feature-compliant with the vanillajs implementation, note that no rules for the framework itself.

TLDR: Think of this benchmark as of a drag racing where everyone will do the craziest turning to his car to win the race, not that all of them will drive on a stock Mustang.

[ClassicOldSong]

Still the name is not so accurate and the drag race points frameworks to the narrow end - people take this benchmark as gold standard for a framework is good or bad nowadays...

[syduki]

people take this benchmark as gold standard

That was a strong saying. To my knoweledge the reality is far away from that. I personally didn't saw so such evidence in the wild, where everyone is pointing to and praising this benchmark, except only a tiny fraction of people, most of them being focused more on frameworks performance and less on developer experience. And since you have mentioned Solid, I will say that it is the one of the few frameworks where every second article in the internet is pointing to this benchmark as to a balls measurement experiment. But if you are right and there are really some people which consider this benchmark as gold standard then we must only feel empathy for their ignorance.

[ClassicOldSong]

To my knoweledge the reality is far away from that.

That's true to framework users, but not authors. This benchmark means more to framework authors than actual end developers. Since end users don't care how frameworks are implemented, they might have no idea they're using innerHTML or not, but whether their knowledge about a framework can be migrated to somewhere else matters. That's why this benchmark may mislead framework authors to a narrow end matters.

[syduki]

Indeed, that's for developers, or if parroting your words - the people. A developer is always an user of some piece of technology, as JS in this very case, and occasionally may be also an author of a JS framework. Many authors of JS frameworks were (and maybe still are) humble users of some other mainstream frameworks, and I bet most of them know how those frameworks are implemented, while maybe not even contributed to them, moreover, I am pretty sure that majority of them got the inspiration there for their own framework, just judging by how many reincarnations of the most popular one are here. The user you are picturing here looks more like a noob, and considering that these benchmarks are tied at this moment to a very specific environment, the "knowledge about a framework can be migrated to somewhere else" is the last thing that should bother them. Any other serious user having such thoughts should be educated enough to understand the limits of a certain framework, at that point he may decide to become an author, ask for help or helping other authors to fix that issue. Here, I see your issue is reduced to "using innerHTML or not", which is an issue only in a half of this benchmark, and sadly, does not resonate at all with the main title. Being so focused on that you forgot about your concern regarding "content updating speed", which is another half of this benchmark, where weights should be of greatest concern, considering that currently the situation there is worse, since those measurements are now in fact for the "delegated click" event not the "updating speed" itself.

[ClassicOldSong]

A developer is always an user of some piece of technology, as JS in this very case, and occasionally may be also an author of a JS framework

I think there're some survivorship bias happing here: there're plenty of developers that won't even try understanding the technology at all, they just do developing for a living, as they stay "noob" forever, they still can solve problems most non-developer people have. And as all participants in this thread all contributed to this benchmark, I'd say we already stepped out of junior developers and might see the world different. But we're not talking about this problem here.

Here, I see your issue is reduced to "using innerHTML or not", which is an issue only in a half of this benchmark, and sadly, does not resonate at all with the main title.

Actually this is my main concern with this issue, the "content updating speed" one happens to be magnified by the newly added weight so I mentioned it. Maybe I should open another issue for that one.