"deno ast script.ts"

[ry]

Should print to stdout a JSON blob representing the typescript AST.

[kitsonk]

The TypeScript AST is quite a different beast than something that is aligned to an ES AST (because the TypeScript parser is closer aligned to Roslyn than it is to what ES thinks of as the AST).

What is the use case, as that might determine things.

https://astexplorer.net/ is a good reference that would demonstrate what a dump would look like and includes the TypeScript AST as well as other parsers, and you can see that there is a big difference between a TypeScript AST and ES based ASTs like acorn.

[ry]

TS also can output AST for JS. (As far as I can remember.)

I want this because

1. It's useful for generating docs.
2. It invites cool projects.
3. We have the ability to do this with relative ease, so we might as well surface it.

[kitsonk]

It can, but it is the Roslyn-like AST, not compatible with the traditional ES ASTs.

For example, given this code:

/**
 * Example of JSDOC
 *
 * @param foo Something foo
 */
function bar(foo) {
  console.log(foo);
}

bar("hello world");

You would get the following from TypeScript:

{
  "pos": 0,
  "end": 122,
  "flags": 0,
  "kind": 279,
  "text": "/**\n * Example of JSDOC\n *\n * @param foo Something foo\n */\nfunction bar(foo) {\n  console.log(foo);\n}\n\nbar(\"hello world\");\n",
  "bindDiagnostics": [],
  "languageVersion": 6,
  "fileName": "astExplorer.tsx",
  "languageVariant": 1,
  "isDeclarationFile": false,
  "scriptKind": 4,
  "pragmas": {},
  "referencedFiles": [],
  "typeReferenceDirectives": [],
  "libReferenceDirectives": [],
  "amdDependencies": [],
  "hasNoDefaultLib": false,
  "statements": [
    {
      "pos": 0,
      "end": 100,
      "flags": 0,
      "parent": "[Circular ~]",
      "kind": 239,
      "jsDoc": [
        {
          "pos": 0,
          "end": 58,
          "flags": 0,
          "parent": "[Circular ~.statements.0]",
          "kind": 291,
          "tags": [
            {
              "pos": 30,
              "end": 56,
              "flags": 0,
              "parent": "[Circular ~.statements.0.jsDoc.0]",
              "kind": 299,
              "tagName": {
                "pos": 31,
                "end": 36,
                "flags": 0,
                "parent": "[Circular ~.statements.0.jsDoc.0.tags.0]",
                "escapedText": "param"
              },
              "name": {
                "pos": 37,
                "end": 40,
                "flags": 0,
                "parent": "[Circular ~.statements.0.jsDoc.0.tags.0]",
                "escapedText": "foo"
              },
              "isNameFirst": true,
              "isBracketed": false,
              "comment": "Something foo"
            }
          ],
          "comment": "Example of JSDOC"
        }
      ],
      "modifierFlagsCache": 536870912,
      "name": {
        "pos": 67,
        "end": 71,
        "flags": 0,
        "parent": "[Circular ~.statements.0]",
        "escapedText": "bar"
      },
      "parameters": [
        {
          "pos": 72,
          "end": 75,
          "flags": 0,
          "parent": "[Circular ~.statements.0]",
          "kind": 151,
          "name": {
            "pos": 72,
            "end": 75,
            "flags": 0,
            "parent": "[Circular ~.statements.0.parameters.0]",
            "escapedText": "foo"
          }
        }
      ],
      "body": {
        "pos": 76,
        "end": 100,
        "flags": 0,
        "parent": "[Circular ~.statements.0]",
        "kind": 218,
        "multiLine": true,
        "statements": [
          {
            "pos": 78,
            "end": 98,
            "flags": 0,
            "parent": "[Circular ~.statements.0.body]",
            "kind": 221,
            "expression": {
              "pos": 78,
              "end": 97,
              "flags": 0,
              "parent": "[Circular ~.statements.0.body.statements.0]",
              "kind": 191,
              "expression": {
                "pos": 78,
                "end": 92,
                "flags": 0,
                "parent": "[Circular ~.statements.0.body.statements.0.expression]",
                "kind": 189,
                "expression": {
                  "pos": 78,
                  "end": 88,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression.expression]",
                  "escapedText": "console"
                },
                "name": {
                  "pos": 89,
                  "end": 92,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression.expression]",
                  "escapedText": "log"
                }
              },
              "arguments": [
                {
                  "pos": 93,
                  "end": 96,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression]",
                  "escapedText": "foo"
                }
              ]
            }
          }
        ]
      }
    },
    {
      "pos": 100,
      "end": 121,
      "flags": 0,
      "parent": "[Circular ~]",
      "kind": 221,
      "expression": {
        "pos": 100,
        "end": 120,
        "flags": 0,
        "parent": "[Circular ~.statements.1]",
        "kind": 191,
        "expression": {
          "pos": 100,
          "end": 105,
          "flags": 0,
          "parent": "[Circular ~.statements.1.expression]",
          "escapedText": "bar"
        },
        "arguments": [
          {
            "pos": 106,
            "end": 119,
            "flags": 0,
            "parent": "[Circular ~.statements.1.expression]",
            "kind": 10,
            "text": "hello world"
          }
        ]
      },
      "modifierFlagsCache": 536870912
    }
  ],
  "endOfFileToken": {
    "pos": 121,
    "end": 122,
    "flags": 0,
    "parent": "[Circular ~]",
    "kind": 1
  },
  "nodeCount": 22,
  "identifierCount": 6,
  "identifiers": {},
  "parseDiagnostics": [],
  "path": "astExplorer.tsx",
  "resolvedPath": "astExplorer.tsx",
  "originalFileName": "astExplorer.tsx",
  "imports": [],
  "moduleAugmentations": [],
  "ambientModuleNames": []
}

Where as you would get the following from acorn:

{
  "type": "Program",
  "start": 0,
  "end": 122,
  "body": [
    {
      "type": "FunctionDeclaration",
      "start": 59,
      "end": 100,
      "id": {
        "type": "Identifier",
        "start": 68,
        "end": 71,
        "name": "bar"
      },
      "expression": false,
      "generator": false,
      "params": [
        {
          "type": "Identifier",
          "start": 72,
          "end": 75,
          "name": "foo"
        }
      ],
      "body": {
        "type": "BlockStatement",
        "start": 77,
        "end": 100,
        "body": [
          {
            "type": "ExpressionStatement",
            "start": 81,
            "end": 98,
            "expression": {
              "type": "CallExpression",
              "start": 81,
              "end": 97,
              "callee": {
                "type": "MemberExpression",
                "start": 81,
                "end": 92,
                "object": {
                  "type": "Identifier",
                  "start": 81,
                  "end": 88,
                  "name": "console"
                },
                "property": {
                  "type": "Identifier",
                  "start": 89,
                  "end": 92,
                  "name": "log"
                },
                "computed": false
              },
              "arguments": [
                {
                  "type": "Identifier",
                  "start": 93,
                  "end": 96,
                  "name": "foo"
                }
              ]
            }
          }
        ]
      }
    },
    {
      "type": "ExpressionStatement",
      "start": 102,
      "end": 121,
      "expression": {
        "type": "CallExpression",
        "start": 102,
        "end": 120,
        "callee": {
          "type": "Identifier",
          "start": 102,
          "end": 105,
          "name": "bar"
        },
        "arguments": [
          {
            "type": "Literal",
            "start": 106,
            "end": 119,
            "value": "hello world",
            "raw": "\"hello world\""
          }
        ]
      }
    }
  ],
  "sourceType": "module"
}

[kitsonk]

For the record, I think it is a good idea.

The other bit, is that the compiler has a decent API for digesting the JSDoc, but it is a side loading thing where you have to query the trivia on each node to see if there is some parsed JSDoc. So I think there are some enhancements for the use cases above to improve on the structure of the AST for documentation generation purposes, or instead of really focusing on a "pure" AST, we actually walk the AST, outputting a JSON "documentation" structure, worrying about the "surface" of the modules integrated with the inline documentation, instead just dumping loads of internals that won't really be useful.

[ry]

The TS AST looks great to me.

[axetroy]

TS AST no need to compatible with ES AST.

and consider adding AST API for deno? or maybe put it into deno_std is better.

[jamiebuilds]

ESTree is a little bit more of a shared standard and the TypeScript ESLint team has a node package that can generate a TypeScript+JSX extension of it which also exports type definitions for all of the AST nodes.

[kitsonk]

Yeah, I am still torn a bit. Ry and I have gone back and forth a few times offline on this as well, related to how we handle the transpilation. Our chats caused me to raise microsoft/TypeScript#33502 as the biggest problem is that while the circular references in the TypeScript AST are possible, there is no easy way to "rehydrate" that AST to be leveraged with the compiler, which is really what something like this would need to be useful.

While ESTree-like ASTs are certainly more common, some potential for reversibility would be critical to this feature, that at some point we could ingest an AST and feed it to the compiler to emit. I can't see that path easy with going down the ESTree-like route.

[jamiebuilds]

If you would like to explore maintaining a AST-to-AST converter, I can describe the challenges and strategies for making it easier to do. I’ve maintained/contributed to several similar tools.

[nayeemrmn]

@ry Closed in #4500, deno doc --json.

[kitsonk]

No... there is a big difference between the doc output and an AST output.

[bartlomieju]

If we really want this subcommand I can add it in the evening, should be 15-minute work...

@ry thoughts?

[kitsonk]

Personally I think it is valuable, especially if we are referring to swc's AST, which I assume we are. It is estree like and there is a fair amount of tooling to be able to do things with it. It would be more useful exposed as a runtime API, IMO, but still useful as a subcommand.

[bartlomieju]

@kitsonk, yeah, that'd be SWC AST. As for runtime API - sure, but we need to fix a few issue before that (#4781)

[ry]

I'm a bit worried about having to maintain a stable output...

I think it's definitely useful as an internal utility - and I thought it might be useful for external people... but not sure.

[kitsonk]

My advice, let's keep it open for now, and if there is a compelling use case down the road, we do it. There are a couple things on my head but won't have time in the near future.

[axetroy]

I think it would be better to expose the APIs of Typescript

const { typescript } = Deno

const sourceFile = typescript.createSourceFile("script.ts", "// typescript content")

// then do whatever you what

[kitsonk]

@axetroy the compiler sits in another worker... Our structure cloning isn't compliant at the moment, but even then, it would get very expensive very quickly to sync up heavily between the two workers like that. We need to always keep the APIs narrowed and focused between the two.

[littledivy]

This would be a great feature/utility imo. Perhaps, using swc ast parser would be sufficient.

For Example: https://github.com/nestdotland/deno_swc/blob/master/examples/parse.ts

I'd be happy to open a PR for this.

[bartlomieju]

There's really nothing stopping us from exposing deno ast --unstable <file> subcommand. There have been no major changes in the swc_ecma_ast implementation in the past couple months (sans support for new TypeScript features). @ry what do you think?

[bartlomieju]

After more discussion we think that this functionality should be provided by the user land libraries - namely there's deno_swc that provides access to SWC APIs using WASM.

[dsherret]

@bartlomieju that's the right decision. It's better for consumers to be able to run any version of deno while picking or sticking to a version of swc to use (avoids: "I can't upgrade deno yet because my code is not compatible with the latest AST changes"/"this code doesn't work in deno [old version goes here]")

[bartlomieju]

@bartlomieju that's the right decision. It's better for consumers to be able to run any version of deno while picking or sticking to a version of swc to use (avoids: "I can't upgrade deno yet because my code is not compatible with the latest AST changes"/"this code doesn't work in deno [old version goes here]")

As well as it's a lot less code to maintain in Deno repo - the typings alone for SWC are several thousand lines.