AST Workshop: Ideas for format/content/resources/etc.?

[kentcdodds]

I basically expect this to be a hands-on version of my talk

[jamestalmage]

• https://astexplorer.net - how to use it. Easily the single most valuable tool.

• Discuss where to find the various node definitions (estree spec, Babels generated docs, ast-types source).

• jscodeshift - probably the best place to point people to start making use of their new found skills. The learning curve on everything else is higher, and there's a much higher likelihood you will write something useful with it.

• Babel plugin handbook - it's not well advertised last time I looked (many months ago), but it's invaluable if you're writing a plugin. Last I checked (again, many months) there was no complete documentation of the Babel plugin API, so discuss that and where to look.

• transform strategies - Basically every transform needs to answer two questions about a node: "Is this a node I want to transform?", and "How should I transform it."

• tracking state as you walk the tree (and recommend avoiding state if you can)

[bakkot]

Probably not worth confusing people with competing standards, but...

ESTree isn't the only AST format! Take a look at Shift (disclosure: this is from @shapesecurity, including @michaelficarra and myself). It's a lot more sane than ESTree, which makes writing analysis / transformation a lot easier. There isn't nearly as much tooling around it, though.

As an example: ESTree can't represent directives directly, which means that one AST can represent two semantically different programs. E.g., acorn gives the same ASTs for these functions, despite them having different semantics:

function f() {
  "use strict";
  return this;
}

function f() {
  ("use strict");
  return this;
}

As a consequence, projects using ESTree in practice end up extending it, usually in incompatible ways: Babylon, for example, adds a Directive node, whereas esprima adds a non-standard "directive" attribute to ExpressionStatement nodes.

ESTree also distinguishes things which it doesn't make sense to distinguish: for example, ({ 0 : null }) and ({ '0': null }) have different ASTs.

Shift doesn't have these problems.

[kentcdodds]

Interesting, thanks for the tip @bakkot. You're correct. It's probably too much for me to delve into that too much. But I'll probably mention that ESTree isn't the only format.

Thanks a ton for all the tips @jamestalmage 👏

[kentcdodds]

Alright, it's about time that I make an outline of what I'm thinking for this workshop:

Background

Very basic "what are ASTs"

Just so we're all on the same page from the get-go. Explaining that AST stands for Abstract Syntax Tree and is a representation of code in an object. I'll probably first explain it, then show a simple example with ASTExplorer

Why are ASTs important?

I like to "start with why" so we'll talk about why anyone would care about ASTs. What can you do with even a basic knowledge and understanding of what ASTs are. The automation possibilities and entire new capabilities that it opens up to you.

How do we use ASTs?

I'll show the parsers available in ASTExplorer and the available transforms. Then I'll mention that for our workshop, we'll focus on three concepts:

1. Writing an ESLint rule with the espree parser and the ESLint v2 transform
2. Writing a Babel plugin with the babylon6 parser and the babelv6 transform (ideas for the demo(s) and the exercise(s) are welcome!)
3. Writing a codemod with the recast parser and the jscodeshift transform (ideas for the demo(s) and the exercise(s) are welcome!). I hear-tell that Babel may have some nice codemod capabilities. I personally prefer Babel's API over jscodeshift's, and it'd be nice to only have to teach the visitor pattern. Anyone in the know want to advise me? cc @hzoo, @loganfsmyth, @DrewML?

Hands on

I think for each concept we'll have two demos and two exercises. One simple and one advanced. We'll do: Simple Demo -> Simple Exercise -> Advanced Demo -> Advanced Exercise

ESLint Rule

Ideas for the demos and the exercises are welcome! Also, feedback on what the key takeaways should be is also appreciated!

Also, if there's time, it'd be great to talk about:

• testing rules
• creating and using a plugin with your rule

Babel Plugin

Request of the Babel team: Can you give me an idea of the things I should keep in mind with regards to changes in v7? Any changes to the plugin API? Or is it mostly user-facing APIs (.babelrc etc?) If there are (significant) changes, any chance I could get a beta version to base this workshop on? This workshop is primarily for a Frontend Masters workshop I'm doing in June.

Ideas for the demos and the exercises are welcome! Also, feedback on what the key takeaways should be is also appreciated!

Also, if there's time, it'd be great to talk about:

• testing plugins
• adding your plugin to the .babelrc for use in your project

Codemod

Like I said, I'd prefer to use Babel for this if possible. I've never had great experiences with JSCodeshift, but perhaps someone can show me the light?

Ideas for the demos and the exercises are welcome! Also, feedback on what the key takeaways should be is also appreciated! Also, if there's time, it'd be great to talk about:

• testing plugins
• creating a CLI for your codemod... maybe... Probably out of scope though.

Conclusion

We'll wrap up by reviewing the value and power of knowing and understanding JavaScript ASTs and the importance to teach these things to others. The more people know about this, the better things will get for everyone.

Any feedback and ideas are welcome! Thanks!

[hzoo]

Request of the Babel team: Can you give me an idea of the things I should keep in mind with regards to changes in v7

It's basically all internal changes + some AST changes to match up with Flow/ESTree. Unlikely you'll hit anything, maybe that we will require presets to be a function instead of an object.

Writing a codemod with the recast parser and the jscodeshift transform (ideas for the demo(s) and the exercise(s) are welcome!). I hear-tell that Babel may have some nice codemod capabilities. I personally prefer Babel's API over jscodeshift's, and it'd be nice to only have to teach the visitor pattern

I explained it in https://babeljs.io/blog/2016/09/28/6.16.0#new-feature - basically add this to your babelrc/config and install recast in node_modules. Then you can just run babel with babel src -d src kinda deal. Or you can just use jscodeshift as usual but instead of using the jscodeshift api use babel-core to transform it.

{
  "parserOpts": {
    "parser": "recast"
  },
  "generatorOpts": {
    "generator": "recast"
  }
}

Also https://github.com/facebook/jscodeshift/issues/168

testing plugins

https://github.com/babel/babel/blob/master/CONTRIBUTING.md#writing-tests

use babel-helper-plugin-test-runner check out https://github.com/babel/babel-preset-env/blob/master/test

[willklein]

@kentcdodds for an example Babel transform, I've found shorthand properties to be fairly straightforward: https://github.com/babel/babel/tree/7.0/packages/babel-plugin-transform-es2015-shorthand-properties

For a Webpack loader, json-loader is also straightforward: https://github.com/webpack-contrib/json-loader

I liked that both demonstrate the basic APIs, are useful/popular, and easy to understand what they're doing.

[kentcdodds]

Thanks @willklein. I'm not demoing anything with Webpack. I'd prefer to do something novel that also demonstrates some level of practicality if possible.

[bmeck]

Ideas:

• codemod to change all references to __line__ to the current line number.
• codemod to change let that never get reassigned to const (would want to avoid var due to hoisting).
• codemod to change functions that don't use this to arrow functions.

[kentcdodds]

Super ideas @bmeck. Thanks!

[hzoo]

Some codemods from a long time ago as reference (didn't check edge cases):

politify: http://astexplorer.net/#/NLHVda3RTq/6 object property shorthand: http://astexplorer.net/#/sskXYR4icD/1 object method shorthand: http://astexplorer.net/#/vd18PVxJ8J/2 destructuring: http://astexplorer.net/#/zdTZE92PrZ/2 split variable declarations: http://astexplorer.net/#/3sxHbt9I7u/1

[kentcdodds]

Thanks! Those politify and split variable declarations examples are awesome.

[kentcdodds]

Who am I kidding. They're all great! Thanks!

[okonet]

You can check out how we use Jest snapshots for testing transforms: https://github.com/webpack/webpack-cli/tree/master/lib/transformations

[kentcdodds]

Thanks for the feedback friends!