CST explorer?

[conartist6]

I've spent the last few months working on a tool, cst-tokens, that I believe will serve as a replacement for recast.

The idea of cst-tokens is to implement the basics of code transformation without any language or parser-specific logic. This could give AST explorer the ability to allow users to easily write transforms for every single language it has a supported parser for!

To do this each parser would need an associated cst-tokens grammar. This isn't as much of a problem as it sounds like because those grammars don't need define complete parsing logic. Instead a grammar just expresses (in essence) what the valid ways of printing a given AST might be. The cst-tokens engine can then use the grammar to recombine a given AST with its source text while tokenizing and attaching the tokens to the nodes they conceptually belong to. Because of the way the problem is simplified, a single grammar can be used to support every estree-compliant parser at the same time (including Babel which is only sort of compliant).

While all this is cool, it creates some architectural problems. cst-tokens isn't really a parser itself. It doesn't belong in the list of parsers. When it's active though, it needs to decorate/replace the tree that the active parser has generated. I don't know enough about how the AST explorer code is structured yet to understand what this would take to implement.

[conartist6]

I'm going to go ahead and try to build a hook that allows a transformer to replace the parsed AST

[conartist6]

Hmm I'm going to need to update transformerMiddleware. I'll need to know the current parser so that buildTransformer can require not just cst-tokens but the appropriate grammar. Then I'll need to make sure changing the parser also rebuilds the transformer.

[conartist6]

I absolutely love that I got issue #666 for this bit of devilry : )

[conartist6]

It's looking like the greatest difficulty in implementing this will be asynchronicity. The important logic is in the parserMiddleware and tranformerMiddleware, but those two operate completely independently of each other: the parse helper is async source => tree and the transform helper is async source => source. Thus there is no place in the existing system for the transformer to to use the parser's output as part of its input and the challenge of implementing the new functionality will be in defining when the new operations occur.

[conartist6]

I think one of the tricky parts is that in my new workflow there would be not one but four main trees:

• The AST the parser produced
• The CST which cst-tokens has used a grammar to decorate with node.cstTokens
• The tree that has been transformed: it may be part AST and part CST.
• The full and valid CST that results from reapplying the grammar

To accommodate this I imagine making use of the tabs UI element at the top of the tree pane, which likely would mean changing the current JSON tab into a checkbox, which makes sense because all these tabs will show trees which can be rendered as JSON.

[conartist6]

I'm happy to do the work, but I'm very curious to hear from @fkling if this is a direction that the project is interested in going?