search

wasmerio / winterjs

Winter is coming... ❄️
https://winterjs.org/
MIT License
3.05k stars 53 forks source link

16-core (WinterJS) vs. 1-core (Node,Bun) benchmarking is not a benchmarking, it's finger painting #54

Closed uNetworkingAB closed 8 months ago

uNetworkingAB commented 8 months ago

Edit:

never mind, I see now that benchmarks are multi-core vs. single-core so completely pointless to even bother communicating anything re. pref.

Arshia001 commented 8 months ago

@uNetworkingAB yes, we've heard about the bun issue from others as well. At the time when we were running the benchmarks, we didn't know bun was single-threaded by default. I'll keep this issue open until we update the numbers. CC @syrusakbary

uNetworkingAB commented 8 months ago

Nodejs, Bun, Deno all 3 are single-threaded by default it's a core design that should be obvious. How are you going to correct this misinformation? The blog post still holds widely invalid numbers

syrusakbary commented 8 months ago

Hey @uNetworkingAB,

We are testing all runtimes, including Node and Bun with their default settings. Shall they upgrade them, shall the benchmarks be updated. If you want other runtimes to use multiple threads, then I'd suggest prompting that on their own repos.

In general, I'd say that the benchmark that we are currently doing is quite limited (as almost any other benchmark that you will find in the wild). Mainly because we are testing literally sending the most simple response ("hello") through it, which in general measures the throughput of the event loop & http server, not JS or the integration.

The benchmark will be greatly improved by testing a full end-to-end scenario, such as number of renders of React per second, which is probably the way to go for benchmarking. I've created a ticket for it to make sure we follow up on that: #56.

But, to be as clear as possible: we will not limit the benchmark to use only one thread just because other runtimes can't currently handle it properly. We believe that, by default, most runtimes should be able to be run multithreaded, specially for WinterCG environments where the context doesn't need to be shared in between requests.

uNetworkingAB commented 8 months ago

with their default settings. Shall they upgrade them, shall the benchmarks be updated. If you want other runtimes to use multiple threads, then I'd suggest prompting that on their own repos.

So you expect Node.js, a 15 year old single-threaded runtime, to change its default to multi-core in order to properly benchmark against it? You want Node.js, highly used runtime, to change to fit your benchmark rather than take the 5 minutes it takes to set up a Node.js cluster and do apples to apples comparison?

You want the world to fit your benchmark rather than make your benchmark fit the world. Nice confidence, but that's never going to happen

eeshankeni commented 8 months ago

bro sounds like he doesn't wanna do actual benchmarks cause deep inside he knows his claims are half baked at best

AddictArts commented 8 months ago

It is fair to compare default out of the box running. They just needed to be clear about that and what they compared and how. The dev response didn't say the other runtimes needed to change. They said, "shall they", just a question. I don't believe they have the burden to prove anything, they should make any claim clear and how they got it. I don't believe the blog post did that well, so questioning the results was also fair. I don't know if the replies were bun fans, but too often the reaction when bun is shown as less than, it is over the top and looks bad. I'd rather see bun just "walk the walk", show it, let that speak for itself. It is flattering to see bun as the benchmark to compare to, that means something, free advertising.

uNetworkingAB commented 8 months ago

Node.js doesn't even have a server by default. It doesn't listen to any port by default and there is no connecting client by default and no attached response.

Winterjs authors explicitly and manually made the competition use a 16th the CPU-time, by explicitly and manually writing the boiler plate code that achieves single thread response, rather than explicitly and manually picking the built-in Cluster module boiler plate code.

None of this is honest, accurate or motivatable.

It's the equivalence of running a manually gered car against an automatically geared one, only running the manual on first fear then posting general numbers representing the car as a whole.

But I have noticed common sense is a rarity nowadays so I wouldn't be shocked people disagree.