Implement new benchmark test/ci system

[TheLarkInn]

webpack performance test harness

For a while, we had setup a performance benchmark system on CI which would compare the performance of any fix/feature at PR time made to webpack/webpack. However, this was so flakey that we had to get rid of it for PR reliability reasons.

I'd like to rebuild a test/CI system that will run performance benchmarks against PR's that we submit to webpack. This could incorporate any of the systems we've set up in /benchmark in the repository.

Performance is one of the key competitive features in bundlers today, and since we are not making any clear stances so far on native technology bets, I would like to keep performance in a tight feedback loop for the project.

Scenario

• [ ] As a maintainer, I want to see a clear performance impact on every PR made to the repository
• [ ] As a maintainer, I want to block any PR on large performance regressions
• [ ] As a maintainer, I want the CI/PR quality gate to be reliable and something I can trust
• [ ] As a contributor, I want a clear/understandable feedback from the performance benchmarking system when I submit a PR.

[TheLarkInn]

@jakebailey I heard from @rbuckton that you all are working on shoring up the reliability of your benchmark framework for TS. Do you have any referenceable issues we could study to implement/improve our own benchmarking suite.

[jakebailey]

I am still experimenting, but one thing that has helped a little is:

• Following/using https://pyperf.readthedocs.io/en/latest/system.html, e.g. setting up a machine where a core is isolated. But if you want to use more than one core, things get really weird based on scheduling, and I'm working on figuring that one out.
• In general, hide benchmark results that are not "significant"; I added to our perf infrastructure something which performs a U-test like https://pkg.go.dev/golang.org/x/perf/benchstat and hides the results which are unlikely to be "real" in that they are indistiguishable from random chance to p=0.05.

There are some other flags I'm testing, specifically --predictable, which make benchmarks super comparable, but I'm still working to udnerstand the implications of this flag and if it's not a good idea to use.

Also: https://github.com/faster-cpython/ideas/issues/480

[webpack-bot]

This issue had no activity for at least three months.

It's subject to automatic issue closing if there is no activity in the next 15 days.

[alexander-akait]

bump

[evenstensberg]

@jakebailey are you still working on this?

[jakebailey]

Er, performance stuff for TypeScript? Or this issue specifically? I'm not working on anything for webpack, no.

[evenstensberg]

I meant this issue. Thx!

[evenstensberg]

@TheLarkInn I'd like to reserve this for gsoc next year, if that is ok?

[evenstensberg]

@snitin315 needs gsoc label

[webpack-bot]

Issue was closed because of inactivity.

If you think this is still a valid issue, please file a new issue with additional information.

[evenstensberg]

@snitin315 reopen this

[souvikiiest]

Dear @snitin315 After going through the project details and related issue #16827, I am genuinely excited about the prospect of contributing to the development of the Webpack Benchmark Tooling. I have a background in JavaScript, Node.js, CSS, and HTML and C++ which are essential prerequisites for this project.

Thank you for considering my expression of interest. I look forward to the opportunity to contribute to the Webpack Benchmark Tooling project and contribute to the continuous improvement of this fantastic tool.

[utshomax]

I see ! interesting project to work on. Seems simple, yet tricky. would love to contribute here , as @TheLarkInn mentioned it's crucial for a bundler.🙃 cc: @snitin315 , @dhruvdutt

[cxde-Sujal]

Hey @snitin315 , @dhruvdutt After going through the project would like to contribute in the development of Webpack Benchmark Tooling. Having familiarity with the skills required in this project . I look forward for an opportunity and look forward for sincere development.

[c6z3h]

Noticed this was on the list of GSOC projects and I'm keen (at least I'm more familiar with CI part than the other bundler issues!). By the way, do we have a list of common breaking changes that we're trying to prevent with the new CI system?

[KunalKumar-1]

has anyone assigned this issue ? or @c6z3h are you working on this issue ?