Babel: thinking about how to make the benchmark more representative

[hzoo]

Ref https://github.com/v8/web-tooling-benchmark/pull/24#issuecomment-345002411

These could all similarly apply for babylon itself FYI we need the bundled/concat'd uncompiled version

the current benchmark:

This benchmark runs the Babel transformation logic using the es2015 preset on a 196KiB ES2015 module containing the untranspiled Vue bundle. Note that this explicitly excludes the Babylon parser and only measures the throughput of the actual transformations. The parser is tested separately by the babylon benchmark below.

https://github.com/v8/web-tooling-benchmark/blob/21ca9e95cd2bd5c0fb6fee0c2f369922ea308a1a/src/babel-benchmark.js#L11

Right now it only tests an ES2015 module (albeit a 194kb one 👍) but that may not be representative of what the future will be like so we should think about possible changes to this benchmark:

• [x] Since we have deprecated the yearly presets like preset-es2015 we should run with @babel/preset-env now https://github.com/v8/web-tooling-benchmark/blob/21ca9e95cd2bd5c0fb6fee0c2f369922ea308a1a/src/babel-benchmark.js#L12
  • I guess we might want to think about different targets but that just runs less of Babel so not sure how useful that is for a benchmark? (targets: default/ie, current node, current chrome, etc)
• [ ] In a similar way I guess we could have a test for an ES3/ES5 file as a good test of the baseline perf of going through the whole program. (The shortcut Babel could do is just to print the file exactly out if it doesn't find any changes, kinda like engines cheat but we won't do that)
  • I just realized we could just run Babel on the output of the original benchmark since that will be ES5 anyway?
• [ ] The payload should test out other kinds of code that people are writing/using with Babel like
  • ES2017+ and Stage x proposals (we could use Babel itself for this if we bundled all of it untranspiled, but there are probably other projects we could use)
  • JSX/Flow/Typescript
• There are other things like compiling a minified source but people shouldn't be doing that?
  • Babel operates per file so realistically it compiles a lot of smaller files

[bmeurer]

Those are great suggestions, Henry, thanks a lot! The current version is mostly a one-shot prototype. Ideally the benchmark payloads would be created and driven by experts like you, who know how a representative workload for Babel looks like.