Benchmark compile times

[overlookmotel]

Problem

Compile times are something we care about, but we're not measuring them systematically, so we have little visibility over them.

Potential solution

Building the benchmarks is a fairly good analogue for real-world usage where you have a warm compiler cache.

• Run benchmark builds with --timings.
• Parse the HTML report to get total compile time for each benchmark.
• Either:
  1. Make a comment on a PR if they improve / regress. or
  2. Upload them to CodSpeed (see last para of #5).
• Bonus points: Display the graph from the timings report in a comment on any PR for which compile times get significantly better/worse.
• Loads of bonus points: Convert the graph from timings report into an .out file so the graph appears as a faux flamegraph on CodSpeed. That'd give you a before/after diff too. (This is probably way too much work, and not worthwhile).

[Boshen]

https://www.val.town/ seems like an easy way to build this.

[Boshen]

Can leverage https://github.com/benchmark-action/github-action-benchmark

Maybe an extra tracking CI for compiling the whole oxc crate, for downstream users.

[Boshen]

Here's what I would like to do:

In transform-ci. Setup https://github.com/benchmark-action/github-action-benchmark

It will track

• oxc crate compile time
• oxc transform
  • wallclock time
  • max rss
  • cpu cycles
  • compiled binary size

Once this is done, copy it over to Rolldown as well.

We may end up not using the github action, and use a custom solution instead https://github.com/web-infra-dev/rspack-ecosystem-benchmark

[overlookmotel]

You want to abandon CodSpeed?

[Boshen]

For the record, I have some setup up here https://github.com/oxc-project/monitor-oxc/blob/b66ad58fa2309bbd3a0c7aec00937c56db23daad/.github/workflows/ci.yml#L199-L217

I may end up putting this in the main repo, but due to how tedious it is to work on github actions, I'm going to abandon this and focus time else where.