Closed rniwa closed 1 year ago
rniwa
commented
2 years ago We currently use the following set of ToDoMVC examples:
Flight
I think it's time to finally retire EmberJS-Debug. What else should be added or retired for v3?
addyosmani
commented
2 years ago Thanks for kicking off this thread! Some drive-by thoughts on the list:
In terms of who should own the updating, I would be happy for @kyliau from our team to assist here (also happy to spread the load if folks would prefer)
rniwa
commented
2 years ago Do we want to take a data-driven (e.g. look at adoption levels) look at what's currently popular to guide our decision making for what to drop? e.g. may be worth checking on Backbone and Inferno
It would be good if we can do data-driven look. Do you have any suggestions on how to gather such data?
rniwa
commented
2 years ago For the scope of this round of updates, do we only want to update or replace, or also add? (given the mention of v3). e.g. if adding, I would probably suggest we include Svelte and a few meta-framework implementations.
We should definitely consider adding as well. It doesn't seem like Svelte has a TodoMVC example. Do you have any demo app in mind that we can make use of?
kyliau
commented
2 years ago Do we want to take a data-driven (e.g. look at adoption levels) look at what's currently popular to guide our decision making for what to drop? e.g. may be worth checking on Backbone and Inferno
It would be good if we can do data-driven look. Do you have any suggestions on how to gather such data?
Our team has some experience working with HttpArchive, which is a public dataset on web technologies. I'm more than happy to help gather such data.
addyosmani
commented
2 years ago We should definitely consider adding as well. It doesn't seem like Svelte has a TodoMVC example. Do you have any demo app in mind that we can make use of?
SGTM. The Svelte team maintain a version at https://github.com/sveltejs/svelte-todomvc. It looks like it hasn't been updated in some time, but we could probably revise it to use the latest version and check it continues to be representative of current Svelte usage patterns.
It would be good if we can do data-driven look. Do you have any suggestions on how to gather such data? Our team has some experience working with HttpArchive, which is a public dataset on web technologies. I'm more than happy to help gather such data.
👍🏻 to leaning into HTTP Archive for these insights. While it has some known limitations, I think it would give us a reasonable enough proxy for decision making purposes here.
rniwa
commented
2 years ago We should definitely consider adding as well. It doesn't seem like Svelte has a TodoMVC example. Do you have any demo app in mind that we can make use of?
SGTM. The Svelte team maintain a version at https://github.com/sveltejs/svelte-todomvc. It looks like it hasn't been updated in some time, but we could probably revise it to use the latest version and check it continues to be representative of current Svelte usage patterns.
Hm... it doesn't have a license. Can we ask them to add licensing terms?
rniwa
commented
2 years ago Cross posting here. Including Polymer might be useful since it's still used by YouTube and it makes extensive use of web component features, which are used on many popular websites these days.
addyosmani
commented
2 years ago Hm... it doesn't have a license. Can we ask them to add licensing terms? Good catch! I've requested that Rich add a LICENSE file to their repo.
Cross posting here. Including Polymer might be useful since it's still used by YouTube and it makes extensive use of web component features, which are used on many popular websites these days.
I would be open to this. My understanding is the Polymer team recommend using https://lit.dev/ (same team) these days but can find out if YouTube plans on sticking with the old version or moving over to this library soon.
bgrins
commented
2 years ago We'd definitely like web components represented in the variants. Discussing with @juliandescottes it seems like Lit would be a good default choice in terms of a library (and of course a vanilla implementation would be good to consider). But interested to see anything Keen is able to find in terms of data.
addyosmani
commented
2 years ago As an update on YouTube: they will be continuing to use Polymer (to one degree or another) for another few years. My two cents would be that we aim to either include Lit (with a few to stressing similar workloads), Lit + Polymer, just Polymer or some other representation of web component workloads.
kyliau
commented
2 years ago Here's a snapshot of the httparchive.technologies.2022_05_12_{desktop, mobile} table showing # origins using JS framework / library of interest:
juliandescottes
commented
2 years ago It's great to have this data, thanks for sharing!
Overall, since we want to include more than just Todo MVC apps here, I wonder if we should be careful about the similarities of the frameworks we pick for TodoMVC? (exaggerating) If all popular frameworks are all virtual dom based, and mostly differ from a developer ergonomics point of view, the benchmark would strongly be biased in favor of a single family of frameworks.
To take a practical example, in the current list there is React/React-redux/Preact. With speedometer 2, was it valuable to have those 3, or did they roughly regress/improve at the same time?
I think it's still healthy to have a few representatives of each framework family, but this might be worth keeping in mind when reviewing the final selection.
shvaikalesh
commented
2 years ago We currently use the following set of ToDoMVC examples:
I believe we also use Flight?
What else should be added
For the JSC team, it would be very much useful to have subtests capturing modern JS features like:
Proxy;BigInt.+1 on web components.
retired for v3
Definitely Inferno, Elm, Flight. Those were quite niche when Speedo2 was created, and now they are plain dead judging by latest release and weekly downloads on npm:
rniwa
commented
2 years ago So it looks like we can retire EmberJS-Debug, Inferno, Elm, and Flight:
And we somehow want to add the following frameworks but they don't have ToDoMVC examples:
kyliau
commented
2 years ago Nuxt and Next.js are meta-frameworks for Vue and React, respectively. Do we want to consider "framework families" like @juliandescottes suggested?
It might be worthwhile to differentiate library vs framework. There are many ways to build an application with a library, but frameworks like Next.js and Nuxt often offer a prescribed way to build things in a more efficient manner (bundling, code splitting, minification, etc out of the box).
On another note, AngularJS has been officially deprecated. Although it's still popular, I think we should just drop it in favor of Angular2 since they belong in the same "family".
rniwa
commented
2 years ago What's the angular mentioned in https://github.com/WebKit/Speedometer/issues/3#issuecomment-1150492924? Is it Angular 2? or AngularJS. We should pick whichever is most popular.
kyliau
commented
2 years ago It's Angular (also known as Angular-Typescript or Angular 2). Long term support (LTS) for AngularJS ended on December 31, 2021.
On your list, we can cross out AngularJS and keep Angular2-TypeScript :)
rniwa
commented
2 years ago It's Angular (also known as Angular-Typescript or Angular 2).
That is an evidence for only including Angular 2.
Long term support (LTS) for AngularJS ended on December 31, 2021.
Just for completeness, I don't think whether something is deprecated or actively maintained is relevant so long as its use is prevalent.
kyliau
commented
2 years ago Just for completeness, I don't think whether something is deprecated or actively maintained is relevant so long as its use is prevalent.
I agree with this in a broad sense, but we ought to keep in mind that the deprecated status of a library or framework likely means that some of the best practice guidelines might no longer be applicable, the tool to build the application might be out of date / incompatible with the runtime, and it'll increase our maintenance burden down the road.
I think it's down to the cost-benefit analysis of prevalence vs maintenance.
For jQuery, even if it's getting less popular, it makes sense to include it because it's just so prevalent. For AngularJS, the adoption is much lower compared to jQuery, and there is a similar successor (Angular) in place, so it's likely not worth the extra burden.
rniwa
commented
2 years ago Just for completeness, I don't think whether something is deprecated or actively maintained is relevant so long as its use is prevalent.
I agree with this in a broad sense, but we ought to keep in mind that the deprecated status of a library or framework likely means that some of the best practice guidelines might no longer be applicable, the tool to build the application might be out of date / incompatible with the runtime, and it'll increase our maintenance burden down the road.
That's irrelevant. So long as a library or a framework is widely used, we need to figure out a way to incorporate into our benchmark. Maintenance cost of benchmark is just about the lowest priority.
rniwa
commented
2 years ago Existing:
Want to add:
bgrins
commented
2 years ago Nuxt and Next.js are meta-frameworks for Vue and React, respectively.
...
There are many ways to build an application with a library, but frameworks like Next.js and Nuxt often offer a prescribed way to build things in a more efficient manner (bundling, code splitting, minification, etc out of the box).
I agree with what @kyliau is saying here, and that makes me think that these frameworks don't really make sense in the TodoMVC set. They're widely adopted enough that some of their common patterns and output make sense to test in speedometer, but I think it would be more tailored towards SSG, or rendering & interacting with a particular flavor of site that leverages their features (blog, ecommerce-type page, scrolling marketing pages, etc). IOW writing a new purely client side TodoMVC implementation doesn't capture what differentiates them from the underlying framework and makes them interesting to test.
vprelovac
commented
1 year ago Couple of things to consider:
By testing JS frameworks, the benchmark measures the level of engine's adaptation to performance issues arisen from poorly coding the framework. In other words, some engines can achieve better performance by specifically targetting poorly written code in those frameworks.
Choosing just one simple practical implementation (ToDoMVC) drastically reduces the breadth and scope of actual JS functionality that the test ends up measuring. Such narrow scope can then be easilly optimized using LTO/PGO, resulting in a engine fast at doing a very small subset of JS while potentially slower at anything outside of that scope. The web can not be approximated by a simple todo app.
Instead perhaps focus on benchmarking atomic level JS functions, that all these frameworks will be using anyway.
Same applies for benchmarking HTML and CSS. Benchmark atomic functionality.
Final score is a combination of HTML, CSS and JS atomic functionality benchmarks.
bgrins
commented
1 year ago Alright, I think this can close now. I landed the change to remove relevant tests in #49. Spun out a separate task to update frameworks for existing tests into #50, and to add the new tests in #51 and #52.
@vprelovac yes, one of the major goals for the project is to reflect more real-world workloads beyond todomvc, but we're carrying over some of the tests from Speedometer 2 as a starting point. Re "atomic" - our goal is to benchmark end to end journeys (outlined in https://github.com/WebKit/Speedometer#what-is-speedometer), since we think that's the best way to make improvements to the benchmark show up for actual people using browsers. There's good background on the motivation for that and challenges with testing single functions in https://webkit.org/blog/3395/speedometer-benchmark-for-web-app-responsiveness/
Update ToDoMVC to use the latest versions of frameworks.