codemodsquad / astx

Super powerful structural search and replace for JavaScript and TypeScript to automate your refactoring
MIT License
92 stars 6 forks source link

Remove early initialization of nodePathPlugin that causes ast-types Scope to malfunction #21

Closed cebamps closed 4 months ago

cebamps commented 4 months ago

Hi!

First of all, for context: I am trying my hand at code mods and found astx after experimenting some with jscodeshift. This project is great! I love that pattern matching on the AST no longer gets in my way. Thank you :)

I have hit two obstacles trying to use ast-types's Scope functionality on matched NodePaths (see ast-types readme). I figure this is not something you have sought to support so far, but I have still managed to work around my immediate obstacles and have produced a couple of small changes to propose.

This PR is for the first issue, which I am more confident about addressing. It removes fork.use(nodePathPlugin) in babelAstTypes, which I believe is done too early and is also not needed since ast-types already handles that for us. (I will explain the second issue in a comment.)

That early use of nodePathPlugin caused the scope property on NodePath instances to be always null. This is because nodePathPlugin uses scopePlugin, which in turns uses typesPlugin at initialization time -- specifically the namedTypes object it provides. But when scopePlugin is initialized too early, namedTypes is still empty: it only gets populated when the definitions of typesPlugin are finalized, which ast-types gets in the right order.

I could not find other parts of ast-types that are similarly sensitive to ordering, so I understand this is not much of an issue for astx until it supports scopes.


This PR first adds a test that witnesses the scope issue, then fixes the broken test by removing the fork.use(nodePathPlugin) call.

The test is a little odd because it checks that scope is non-null, though astx omits scope when redefining the NodePath interface, so I have a type mismatch. Adding scope to that re-definition looks like a big change that I can't really see as a good idea from where I stand.

cebamps commented 4 months ago

The second obstacle I found that stands in the way of supporting scope is in how Babel's type definitions do not fully match the expectations set by the ast-types scopePlugin. But that may be a limitation of ast-types itself, so I am unsure about creating a PR... You can find my fix here; let me know if this seems appropriate to submit: https://github.com/cebamps/astx/compare/fix/ast-types-init...cebamps:astx:fix/ast-types-scope

The issue is that the ast-types scopePlugin implicitly relies on the Pattern type being a supertype of Identifier (see assertion in scopePlugin, and the Identifier definition in ast-types), but @babel/types distinguishes Pattern and PatternLike, with Babel's Identifier only having the latter as a supertype.

Therefore, ast-types scope lookups simply fail the assertion in a simple case I tried:

Example test case using scope ```ts import { NodePath as AstTypesNodePath } from 'ast-types/lib/node-path' import { Astx } from '../../src' import { astxTestcase } from '../astxTestcase' import dedent from 'dedent-js' astxTestcase({ file: __filename, input: dedent` const x = 1, y = 2; function foo() { const y = 3; const target = x + y; } `, expected: dedent` const x = 1, y = 2; function foo() { const y = 3; const target = 1 + 3; } `, parsers: ['babel', 'babel/tsx'], astx: ({ astx }) => { for (const { $a, $b } of astx.find`const target = $a + $b`) { inlineNaively($a) inlineNaively($b) } }, }) /** kludge to gain access to scope */ type AstxWithScope = Astx & { path: AstTypesNodePath } // inspired by jscodeshift function getDeclarator(a: AstxWithScope) { const name = a.path.value.name const bindings = new Astx( a.context, a.path.scope?.lookup(name)?.getBindings()[name] ) return bindings .closest((astx) => astx.node.type === 'VariableDeclarator') .at(0) } function inlineNaively(a: Astx) { const val = getDeclarator(a as AstxWithScope).path.get('init').value a.replace(val) } ```

Adding an extra relationship in babelAstTypes by adding an extra line def('Identifier').bases('Pattern'); makes it work, but I suspect this kind of ad-hoc definition tweak is not quite right for astx because it introduces a difference with the Babel type hierarchy. Perhaps it should instead fall on ast-types to make a few less assumptions about what the Pattern type contains?

jedwards1211 commented 4 months ago

Wow thanks for the write-up! I had experimented with trying to get scopes to work briefly once but got sidetracked before I could debug what was going wrong.

I've been thinking about implementing my own scope lookups, though it would obviously take quite a bit of work. In the past I have found ast-types to miss some bindings, at least with Flow type parameters. Also do you know if ast-types scopes continue to work after ast modifications? I haven't read the relevant code in awhile. I've definitely run into some issues with Babel's own scope implementation and drastic modifications.

And yeah the fact that the code in ast-types isn't perfectly tailored to Babel's AST is concerning. (I think I've seen similar issues with Recast).

Ultimately it would be nice if the following uses scope lookups:

const { $foo } = astx.findImports`import $foo from 'foo'`

const calls = astx.find`$foo($$args)` // only matches if the callee is the identifier bound by the import declaration!
jedwards1211 commented 4 months ago

I love that pattern matching on the AST no longer gets in my way.

Thanks! Let me know if you run into any pain points. There definitely are some, for example there's no pattern that will match all possible class declarations right now.

jedwards1211 commented 4 months ago

I suspect this kind of ad-hoc definition tweak is not quite right for astx because it introduces a difference with the Babel type hierarchy

It's probably not a huge deal, I don't think .def('Identifier').bases('Pattern') would break traversal in any way. If anyone were usingnamedTypes.Pattern.check(node)(so far unlikely) it would affect that. But the scope bindings may be inaccurate depending on howPatternLikeandPattern` differ.

I think the way ast-types mashes multiple AST schemas together in its default defs is problematic...that's why I decided to build my own defs from @babel/types. And the scope code is obviously not going to be agnostic to the AST schema. It's probably better to copy ast-types' scope plugin code and tailor it precisely to Babel's AST.

cebamps commented 4 months ago

Thanks for taking the time to respond!

Also do you know if ast-types scopes continue to work after ast modifications?

I played around a bit and found you can call path.scope.scan(true) to force a re-scan of a path's scope after modifying the tree. I don't know how reliable that is, but without it you can get some broken results. Here's a demo using recast in AST explorer: https://astexplorer.net/#/gist/bfa306fe1b7da6f493cd166a99ea1d9e/6aae388fb4e75c8d872400b0ea1ddd4e78b5051a (which I don't trust uses the latest ast-types, after checking out its current yarn.lock)

I've definitely run into some issues with Babel's own scope implementation and drastic modifications.

I'm not far enough yet into my own endeavors to know if I will bump into these. Wish me luck, I guess! (Edit: oh, Babel's own scope implementation! I had misread that. I had not found out about that yet.)

Thanks! Let me know if you run into any pain points. There definitely are some, for example there's no pattern that will match all possible class declarations right now.

Thank you as well! My plan is to automate a refactoring of quite a few React function components, that requires me to move some value declarations and TypeScript annotations between scopes. Not sure how far I will get with it, but I will let you know if I run into astx pain points.

I think the way ast-types mashes multiple AST schemas together in its default defs is problematic...that's why I decided to build my own defs from @babel/types. And the scope code is obviously not going to be agnostic to the AST schema. It's probably better to copy ast-types' scope plugin code and tailor it precisely to Babel's AST.

Makes sense! I found a conversation you had with the Babel folks while researching all this; it's quite amazing how these two pieces end up fitting together! But i can see how this stretches the scope tracking from ast-types a bit too far :). I'll try to update this discussion if I run into any blockers with the scope plugin in my other branch.

jedwards1211 commented 4 months ago

I played around a bit and found you can call path.scope.scan(true) to force a re-scan of a path's scope after modifying the tree.

Nice. I'm tempted to make custom scope lookup code that does a fresh scan on each lookup, so that it always returns the right answer. It would be slower though. It could at least bail out when it finds the target identifier, but would have to scan everything if the target identifier doesn't exist.

Another alternative would be that as long as you use astx API to modify the AST, the scope would be reliable. But that might be too inconvenient...

jedwards1211 commented 4 months ago

I went ahead and added .def('Identifier').bases('Pattern'), I think it will be low risk for now!

jedwards1211 commented 4 months ago

:tada: This PR is included in version 3.0.0-beta.24 :tada:

The release is available on:

Your semantic-release bot :package::rocket: