Benchmarks for Async Web Tooling

[bmeck]

This was talked about in the feedback meeting, but https://github.com/v8/web-tooling-benchmark currently is only setup to test a minimal subset of workflows that do not use asynchronous tasks. That is the reason that webpack is not in the benchmark suite there. It would be great if we could setup some generic benchmarking system to use against real applications. I did mention that GoDaddy has an open source build system that has a component that seems to apply.

We could look at modifying Carpenter to add any metrics gathering the group thinks as relevant or any other parts of our system if that seems desirable. I'm not sure if we have specific benchmark open source sites in mind but I can see if we can also open source some more of ours but that would take some significant time.

[mathiasbynens]

In the Web Tooling Benchmark, we avoid:

• Node-specific APIs, to ensure the benchmark can run in the various standalone JS shells
• disk I/O, as it skews the results (on non-SSD drives, it would otherwise become an I/O benchmark)

We’re open to testing asynchronous tasks, as long as they don’t fall under the above, and provided we find a way around the fact that the microtask queue is not handled consistently across JS shells.

[m-leitch]

Given IO tends to be where things get "real", can you simply enforce a reference standard (whether IOPS+latency or SSD+NVMe)? This would assume these additional workloads would have value. :-)

[bmeck]

@mathiasbynens since IO is part of Node's bottlenecks sometimes I think we would want it to some extent, even if skewed / needing to plaec timers on start/end of every turn of the event loop. Otherwise we are just testing JS running speed, not any IO optimizations as well. I'm not sure how complex real world apps would be easily ported to not use Node APIs but that seems a fine goal if it is doable.

[mhdawson]

I agree that having something that covers IO optimization is good as well.