Implementing continuous benchmarking with CodSpeed

[MarkMcCulloh]

https://codspeed.io

It's a relatively new SaaS, but I like their direction. Seems to work well in https://github.com/oxc-project/oxc

• Continuous tracking of perf
• Builtin system to block perf regressions in PR
• Instruments the runtime so that benches can be sampling based instead of time based. This solves the CI variance problem
• Only works in x64 linux, but that all we're doing right now anyways
• Should integrate easily with existing benchmarks
  • Not sure if we'll have to rewrite the tests so that it's not mostly executed in a child process
• Has OSS pricing (free)

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 60 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!