Revisit JavaScript build tooling

[hancush]

Our current toolchain for transpiling JavaScript is to add a custom precompiler to Django Compressor calling Browserify with a Babelify transform step, generally using offline compression to bundle the resulting scripts as static assets. This works ok, but some less than ideal features include:

• Opacity re: how JavaScript configs (package.json, .bablerc) are used, if at all
• Different behavior for identical commands, depending on whether a script is inline or packaged
• Manual identification of third-party plugins that need to be transpiled for browser compatibility

I've discussed each of these at more length in https://github.com/datamade/how-to/issues/149#issuecomment-738339006. I'd like to look at what we'd need to change to break transpilation into its own build step, rather than running it at as a precompiler via Django Compressor.

Original description

Documentation request

There are a lot of config files that go into any mildly complex JavaScript build, and I don't fully understand how they work together and/or affect other pieces of the build, e.g., Browserify and Babel. I'd like to expand our docs to better articulate these relationships so I finally learn how this all works.

[hancush]

One problem I'm having is that it's really opaque how, if at all, Django Compressor uses .babelrc, etc. Follow up question: Will we ever want in-request transpilation of JavaScript, and if not, why do we favor Django Compressor over a straight up JavaScript build step?

[fgregg]

on demand transpilation seems surprising to me too. i don't know why you wouldn't want those be static assets.

[hancush]

Some thoughts:

• I don't really buy the don't support IE arguments, in particular because our work is relatively more likely to be used by government / other legacy system users, so we need to retain transpilation as a build step. To that end, I like Browserify and Babelify (which we use as a precompile step in Django Compressor).
• One big pain point of our current workflow is that it's not super apparent how build configs are used, if at all, by Django Compressor. For instance, inline transformation seems to yield a different result than transforming modules, even though the commands are identical.
• Another pain point is that Babel doesn't transpile plugins by default. This can lead to vexing backwards incompatibility when a handful of plugins target Node but not the browser. Babel can be configured to transpile these dependencies like this: babelify --global --only [ cdi/static /static node_modules/query-string node_modules/split-on-first node_modules/strict-uri-encode ] --presets [ @babel/preset-env @babel/preset-react ] --plugins [ @babel/plugin-proposal-class-properties ]. However, the path toward identifying problematic dependencies is a matter of running the app in IE, identifying the breaking code from the console, and grepping node_modules to identify where the code lives.
  • There is a Webpack loader that targets plugins for transpilation if they target a Node version > one corresponding to ES5. I wonder if something like this exists for Browserify, and if it doesn't, whether we should consider migrating to Webpack (although, the project looks pretty dead on GitHub :-/.)
  • TBH, we could reuse most of that code to identify the dependencies that need transpilation. Simply having a list would be an improvement over guessing and checking.
• One of the nice things about Django Compressor is that it includes a management command we can use to package transpiled files as static assets. This fits into our workflow and existing code organization practices nicely.
• With that said, it also enables some iffy scripting practices, in particular where our scripts interact with data from the Django template. In some ways, I'd like our JavaScript tooling to be a forcing function for better scripting practices, i.e., no inline scripts.

[hancush]

Eureka!

• https://babeljs.io/docs/en/config-files#project-wide-configuration

  New in Babel 7.x, Babel has a concept of a "root" directory, which defaults to the current working directory. For project-wide configuration, Babel will automatically search for a babel.config.json file

• https://github.com/babel/babelify/issues/293

  Plugins specified on the command line are combined with the plugins specified in the nearest babel config file.

[hancush]

Started updates with a default Babel config that will transpile our JavaScript to ES5 compatible syntax and instructions for transpiling problematic dependencies in https://github.com/datamade/how-to/compare/master..hcg/babel-config.

I'd love to find a better solution than guess and check to identify those problem modules. Next time!

[hancush]

I wrote a simple Python CLI that walks a node_modules directory and identifies packages targeting versions of Node that may include ES6 syntax, i.e., break browser compatibility: https://github.com/hancush/python-es6-sniffer

It successfully identified the problematic packages from CA FWD, plus a few others.

I chose this, because switching to Webpack to use the plugin that achieves the same thing seems too heavy for our use case. (One article described Browserify as Lincoln Logs and Webpack as the whole cabin, which I found helpful.)

I'll add guidance on how to use the CLI next time.