oxc-project / oxc

⚓ A collection of JavaScript tools written in Rust.
https://oxc.rs
MIT License
10.63k stars 390 forks source link

`oxc-parser` NPM package return AST as JS object, not JSON #4295

Open overlookmotel opened 1 month ago

overlookmotel commented 1 month ago

At present, NPM package oxc-parser's parse* methods return an object including AST as JSON string.

I propose:

This is a breaking change, and will require bumping the package minor version (i.e. 0.21.0).

This is motivated by oxc-project/backlog#47.

overlookmotel commented 1 month ago

@Boshen and I discussed on video call just now. He's given his 👍 to this.

dalisoft commented 1 month ago

@overlookmotel Supporting TypeScript interfaces for program also would be helpful or using as TypeScript object as const which returns RAW data as type

overlookmotel commented 1 month ago

Hi @dalisoft. Sorry but I don't quite understand what you mean. I'm not much of a TypeScript user, so perhaps that's the reason. Could you possibly elaborate?

dalisoft commented 1 month ago

@overlookmotel

Input

const funcString =
    /**
     *
     * @returns {{status: 'success' | 'error'}}
     */
    (async (req, res) => {
      const var1 = req.headers['X-Telemetry-ID'];
      const var2 = req.cookies['X-User-ID'];

      useEffect(() => {
        let unload;
        const initTelemetry = async () => {
          await xUserConnect(req.params['x-user-id']);

          ({ unload } = await loadTelemetryApp(var1));
        };
        initTelemetry();

        return unload;
      }, [var1, var2]);

      const result = await someExpensiveTask();
      const confirm = await waitToConfirm(req, result);

      const { cookies } = req;
      const { send } = res;
      const queries = req.queries;
      const { compress } = req.queries;
      const { check_zstd } = queries;

      const values = [req.body];
      const foo = {
        bar: req.body.bar
      };
      const template = `${req.params.mailId}`;

      await testMe(queries.elemId);

      shouldSync(req.params);
      await checkAuth(req.headers, req.cookies);

      await itWorks(result, confirm);
      itWorksSync(confirm);

      await (async () => {
        await someSQLTask();
      })();

      return { status: 'success' };
    }).toString();

  const oxcParserResult = await oxcParser.parseAsync(funcString, {
    sourceType: 'unambiguous',
    sourceFilename: `route-${i}.ts`,
    preserveParens: true
  });
  const { errors, comments, program } = oxcParserResult;

  // program is `string` at here, but as this issue propose is using `JS` instead of `JSON` string, i would like to expect `TypeScript types` too

Propose 1 (TypeScript types)

// This is for example, this interface/type could be improved
// with type-based properties like `body` or `properties`
interface ASTNode {
  type:
    | 'Program'
    | 'ExpressionStatement'
    | 'ArrowFunctionExpression'
    | 'CallExpression' /* so on other types */;
  start: number;
  end: number;
  body?: ASTNode[];
}

  const { errors, comments, program } = oxcParserResult;
  // program is `ASTNode` at here for better completion

Propose 2 (TypeScript as const)

ast-tree.ts ```ts const result = { type: 'Program', start: 0, end: 892, sourceType: { language: 'typescript', moduleKind: 'module', variant: 'standard', alwaysStrict: false }, hashbang: null, directives: [], body: [ { type: 'ExpressionStatement', start: 0, end: 892, expression: { type: 'ArrowFunctionExpression', start: 0, end: 892, expression: false, async: true, typeParameters: null, params: { type: 'FormalParameters', start: 6, end: 16, kind: 'ArrowFormalParameters', items: [ { type: 'FormalParameter', start: 7, end: 10, decorators: [], pattern: { type: 'Identifier', start: 7, end: 10, name: 'req', typeAnnotation: null, optional: false }, accessibility: null, readonly: false, override: false }, { type: 'FormalParameter', start: 12, end: 15, decorators: [], pattern: { type: 'Identifier', start: 12, end: 15, name: 'res', typeAnnotation: null, optional: false }, accessibility: null, readonly: false, override: false } ] }, returnType: null, body: { type: 'FunctionBody', start: 20, end: 892, directives: [], statements: [ { type: 'VariableDeclaration', start: 26, end: 102, kind: 'const', declarations: [ { type: 'VariableDeclarator', start: 32, end: 68, id: { type: 'Identifier', start: 32, end: 36, name: 'var1', typeAnnotation: null, optional: false }, init: { type: 'ComputedMemberExpression', start: 39, end: 68, object: { type: 'StaticMemberExpression', start: 39, end: 50, object: { type: 'Identifier', start: 39, end: 42, name: 'req' }, property: { type: 'Identifier', start: 43, end: 50, name: 'headers' }, optional: false }, expression: { type: 'StringLiteral', start: 51, end: 67, value: 'X-Telemetry-ID' }, optional: false }, definite: false }, { type: 'VariableDeclarator', start: 70, end: 101, id: { type: 'Identifier', start: 70, end: 74, name: 'var2', typeAnnotation: null, optional: false }, init: { type: 'ComputedMemberExpression', start: 77, end: 101, object: { type: 'StaticMemberExpression', start: 77, end: 88, object: { type: 'Identifier', start: 77, end: 80, name: 'req' }, property: { type: 'Identifier', start: 81, end: 88, name: 'cookies' }, optional: false }, expression: { type: 'StringLiteral', start: 89, end: 100, value: 'X-User-ID' }, optional: false }, definite: false } ], declare: false }, { type: 'ExpressionStatement', start: 107, end: 324, expression: { type: 'CallExpression', start: 107, end: 323, arguments: [ { type: 'ArrowFunctionExpression', start: 117, end: 308, expression: false, async: false, typeParameters: null, params: { type: 'FormalParameters', start: 117, end: 119, kind: 'ArrowFormalParameters', items: [] }, returnType: null, body: { type: 'FunctionBody', start: 123, end: 308, directives: [], statements: [ { type: 'VariableDeclaration', start: 131, end: 142, kind: 'let', declarations: [ { type: 'VariableDeclarator', start: 135, end: 141, id: { type: 'Identifier', start: 135, end: 141, name: 'unload', typeAnnotation: null, optional: false }, init: null, definite: false } ], declare: false }, { type: 'ExpressionStatement', start: 149, end: 281, expression: { type: 'CallExpression', start: 149, end: 280, arguments: [], callee: { type: 'ParenthesizedExpression', start: 149, end: 278, expression: { type: 'ArrowFunctionExpression', start: 150, end: 277, expression: false, async: true, typeParameters: null, params: { type: 'FormalParameters', start: 156, end: 158, kind: 'ArrowFormalParameters', items: [] }, returnType: null, body: { type: 'FunctionBody', start: 162, end: 277, directives: [], statements: [ { type: 'ExpressionStatement', start: 172, end: 216, expression: { type: 'AwaitExpression', start: 172, end: 215, argument: { type: 'CallExpression', start: 178, end: 215, arguments: [ { type: 'ComputedMemberExpression', start: 191, end: 214, object: { type: 'StaticMemberExpression', start: 191, end: 201, object: { type: 'Identifier', start: 191, end: 194, name: 'req' }, property: { type: 'Identifier', start: 195, end: 201, name: 'params' }, optional: false }, expression: { type: 'StringLiteral', start: 202, end: 213, value: 'x-user-id' }, optional: false } ], callee: { type: 'Identifier', start: 178, end: 190, name: 'xUserConnect' }, typeParameters: null, optional: false } } }, { type: 'ExpressionStatement', start: 225, end: 269, expression: { type: 'ParenthesizedExpression', start: 225, end: 268, expression: { type: 'AssignmentExpression', start: 226, end: 267, operator: '=', left: { type: 'ObjectAssignmentTarget', start: 226, end: 236, properties: [ { type: 'AssignmentTargetPropertyIdentifier', start: 228, end: 234, binding: { type: 'Identifier', start: 228, end: 234, name: 'unload' }, init: null } ] }, right: { type: 'AwaitExpression', start: 239, end: 267, argument: { type: 'CallExpression', start: 245, end: 267, arguments: [ { type: 'Identifier', start: 262, end: 266, name: 'var1' } ], callee: { type: 'Identifier', start: 245, end: 261, name: 'loadTelemetryApp' }, typeParameters: null, optional: false } } } } } ] }, scopeId: null } }, typeParameters: null, optional: false } }, { type: 'ReturnStatement', start: 288, end: 302, argument: { type: 'Identifier', start: 295, end: 301, name: 'unload' } } ] }, scopeId: null }, { type: 'ArrayExpression', start: 310, end: 322, elements: [ { type: 'Identifier', start: 311, end: 315, name: 'var1' }, { type: 'Identifier', start: 317, end: 321, name: 'var2' } ] } ], callee: { type: 'Identifier', start: 107, end: 116, name: 'useEffect' }, typeParameters: null, optional: false } }, { type: 'VariableDeclaration', start: 329, end: 621, kind: 'const', declarations: [ { type: 'VariableDeclarator', start: 335, end: 369, id: { type: 'Identifier', start: 335, end: 341, name: 'result', typeAnnotation: null, optional: false }, init: { type: 'AwaitExpression', start: 344, end: 369, argument: { type: 'CallExpression', start: 350, end: 369, arguments: [], callee: { type: 'Identifier', start: 350, end: 367, name: 'someExpensiveTask' }, typeParameters: null, optional: false } }, definite: false }, { type: 'VariableDeclarator', start: 371, end: 413, id: { type: 'Identifier', start: 371, end: 378, name: 'confirm', typeAnnotation: null, optional: false }, init: { type: 'AwaitExpression', start: 381, end: 413, argument: { type: 'CallExpression', start: 387, end: 413, arguments: [ { type: 'Identifier', start: 401, end: 404, name: 'req' }, { type: 'Identifier', start: 406, end: 412, name: 'result' } ], callee: { type: 'Identifier', start: 387, end: 400, name: 'waitToConfirm' }, typeParameters: null, optional: false } }, definite: false }, { type: 'VariableDeclarator', start: 415, end: 432, id: { type: 'ObjectPattern', start: 415, end: 426, properties: [ { type: 'BindingProperty', start: 417, end: 424, key: { type: 'Identifier', start: 417, end: 424, name: 'cookies' }, value: { type: 'Identifier', start: 417, end: 424, name: 'cookies', typeAnnotation: null, optional: false }, shorthand: true, computed: false } ], typeAnnotation: null, optional: false }, init: { type: 'Identifier', start: 429, end: 432, name: 'req' }, definite: false }, { type: 'VariableDeclarator', start: 434, end: 448, id: { type: 'ObjectPattern', start: 434, end: 442, properties: [ { type: 'BindingProperty', start: 436, end: 440, key: { type: 'Identifier', start: 436, end: 440, name: 'send' }, value: { type: 'Identifier', start: 436, end: 440, name: 'send', typeAnnotation: null, optional: false }, shorthand: true, computed: false } ], typeAnnotation: null, optional: false }, init: { type: 'Identifier', start: 445, end: 448, name: 'res' }, definite: false }, { type: 'VariableDeclarator', start: 450, end: 471, id: { type: 'Identifier', start: 450, end: 457, name: 'queries', typeAnnotation: null, optional: false }, init: { type: 'StaticMemberExpression', start: 460, end: 471, object: { type: 'Identifier', start: 460, end: 463, name: 'req' }, property: { type: 'Identifier', start: 464, end: 471, name: 'queries' }, optional: false }, definite: false }, { type: 'VariableDeclarator', start: 473, end: 499, id: { type: 'ObjectPattern', start: 473, end: 485, properties: [ { type: 'BindingProperty', start: 475, end: 483, key: { type: 'Identifier', start: 475, end: 483, name: 'compress' }, value: { type: 'Identifier', start: 475, end: 483, name: 'compress', typeAnnotation: null, optional: false }, shorthand: true, computed: false } ], typeAnnotation: null, optional: false }, init: { type: 'StaticMemberExpression', start: 488, end: 499, object: { type: 'Identifier', start: 488, end: 491, name: 'req' }, property: { type: 'Identifier', start: 492, end: 499, name: 'queries' }, optional: false }, definite: false }, { type: 'VariableDeclarator', start: 501, end: 525, id: { type: 'ObjectPattern', start: 501, end: 515, properties: [ { type: 'BindingProperty', start: 503, end: 513, key: { type: 'Identifier', start: 503, end: 513, name: 'check_zstd' }, value: { type: 'Identifier', start: 503, end: 513, name: 'check_zstd', typeAnnotation: null, optional: false }, shorthand: true, computed: false } ], typeAnnotation: null, optional: false }, init: { type: 'Identifier', start: 518, end: 525, name: 'queries' }, definite: false }, { type: 'VariableDeclarator', start: 527, end: 546, id: { type: 'Identifier', start: 527, end: 533, name: 'values', typeAnnotation: null, optional: false }, init: { type: 'ArrayExpression', start: 536, end: 546, elements: [ { type: 'StaticMemberExpression', start: 537, end: 545, object: { type: 'Identifier', start: 537, end: 540, name: 'req' }, property: { type: 'Identifier', start: 541, end: 545, name: 'body' }, optional: false } ] }, definite: false }, { type: 'VariableDeclarator', start: 548, end: 585, id: { type: 'Identifier', start: 548, end: 551, name: 'foo', typeAnnotation: null, optional: false }, init: { type: 'ObjectExpression', start: 554, end: 585, properties: [ { type: 'ObjectProperty', start: 562, end: 579, kind: 'init', key: { type: 'Identifier', start: 562, end: 565, name: 'bar' }, value: { type: 'StaticMemberExpression', start: 567, end: 579, object: { type: 'StaticMemberExpression', start: 567, end: 575, object: { type: 'Identifier', start: 567, end: 570, name: 'req' }, property: { type: 'Identifier', start: 571, end: 575, name: 'body' }, optional: false }, property: { type: 'Identifier', start: 576, end: 579, name: 'bar' }, optional: false }, init: null, method: false, shorthand: false, computed: false } ] }, definite: false }, { type: 'VariableDeclarator', start: 587, end: 620, id: { type: 'Identifier', start: 587, end: 595, name: 'template', typeAnnotation: null, optional: false }, init: { type: 'TemplateLiteral', start: 598, end: 620, quasis: [ { type: 'TemplateElement', start: 599, end: 599, tail: false, value: { raw: '', cooked: '' } }, { type: 'TemplateElement', start: 619, end: 619, tail: true, value: { raw: '', cooked: '' } } ], expressions: [ { type: 'StaticMemberExpression', start: 601, end: 618, object: { type: 'StaticMemberExpression', start: 601, end: 611, object: { type: 'Identifier', start: 601, end: 604, name: 'req' }, property: { type: 'Identifier', start: 605, end: 611, name: 'params' }, optional: false }, property: { type: 'Identifier', start: 612, end: 618, name: 'mailId' }, optional: false } ] }, definite: false } ], declare: false }, { type: 'ExpressionStatement', start: 626, end: 655, expression: { type: 'AwaitExpression', start: 626, end: 654, argument: { type: 'CallExpression', start: 632, end: 654, arguments: [ { type: 'StaticMemberExpression', start: 639, end: 653, object: { type: 'Identifier', start: 639, end: 646, name: 'queries' }, property: { type: 'Identifier', start: 647, end: 653, name: 'elemId' }, optional: false } ], callee: { type: 'Identifier', start: 632, end: 638, name: 'testMe' }, typeParameters: null, optional: false } } }, { type: 'ExpressionStatement', start: 660, end: 683, expression: { type: 'CallExpression', start: 660, end: 682, arguments: [ { type: 'StaticMemberExpression', start: 671, end: 681, object: { type: 'Identifier', start: 671, end: 674, name: 'req' }, property: { type: 'Identifier', start: 675, end: 681, name: 'params' }, optional: false } ], callee: { type: 'Identifier', start: 660, end: 670, name: 'shouldSync' }, typeParameters: null, optional: false } }, { type: 'ExpressionStatement', start: 688, end: 730, expression: { type: 'AwaitExpression', start: 688, end: 729, argument: { type: 'CallExpression', start: 694, end: 729, arguments: [ { type: 'StaticMemberExpression', start: 704, end: 715, object: { type: 'Identifier', start: 704, end: 707, name: 'req' }, property: { type: 'Identifier', start: 708, end: 715, name: 'headers' }, optional: false }, { type: 'StaticMemberExpression', start: 717, end: 728, object: { type: 'Identifier', start: 717, end: 720, name: 'req' }, property: { type: 'Identifier', start: 721, end: 728, name: 'cookies' }, optional: false } ], callee: { type: 'Identifier', start: 694, end: 703, name: 'checkAuth' }, typeParameters: null, optional: false } } }, { type: 'ExpressionStatement', start: 735, end: 766, expression: { type: 'AwaitExpression', start: 735, end: 765, argument: { type: 'CallExpression', start: 741, end: 765, arguments: [ { type: 'Identifier', start: 749, end: 755, name: 'result' }, { type: 'Identifier', start: 757, end: 764, name: 'confirm' } ], callee: { type: 'Identifier', start: 741, end: 748, name: 'itWorks' }, typeParameters: null, optional: false } } }, { type: 'ExpressionStatement', start: 771, end: 792, expression: { type: 'CallExpression', start: 771, end: 791, arguments: [ { type: 'Identifier', start: 783, end: 790, name: 'confirm' } ], callee: { type: 'Identifier', start: 771, end: 782, name: 'itWorksSync' }, typeParameters: null, optional: false } }, { type: 'ExpressionStatement', start: 797, end: 854, expression: { type: 'AwaitExpression', start: 797, end: 853, argument: { type: 'CallExpression', start: 803, end: 853, arguments: [], callee: { type: 'ParenthesizedExpression', start: 803, end: 851, expression: { type: 'ArrowFunctionExpression', start: 804, end: 850, expression: false, async: true, typeParameters: null, params: { type: 'FormalParameters', start: 810, end: 812, kind: 'ArrowFormalParameters', items: [] }, returnType: null, body: { type: 'FunctionBody', start: 816, end: 850, directives: [], statements: [ { type: 'ExpressionStatement', start: 824, end: 844, expression: { type: 'AwaitExpression', start: 824, end: 843, argument: { type: 'CallExpression', start: 830, end: 843, arguments: [], callee: { type: 'Identifier', start: 830, end: 841, name: 'someSQLTask' }, typeParameters: null, optional: false } } } ] }, scopeId: null } }, typeParameters: null, optional: false } } }, { type: 'ReturnStatement', start: 859, end: 888, argument: { type: 'ObjectExpression', start: 866, end: 887, properties: [ { type: 'ObjectProperty', start: 868, end: 885, kind: 'init', key: { type: 'Identifier', start: 868, end: 874, name: 'status' }, value: { type: 'StringLiteral', start: 876, end: 885, value: 'success' }, init: null, method: false, shorthand: false, computed: false } ] } } ] }, scopeId: null } } ], scopeId: null } as const; ``` Copy, paste into your IDE and try `result.` and see how works smartly and you'll see all properties as-is (due of how `as const` TypeScript trick works)
overlookmotel commented 1 month ago

Ah ha. Thanks for clarifying. We are working on infrastructure to generate TS type definitions from our internal Rust types. It'll take a little time to do that (not quite as simple as it sounds!), but it is coming.

dalisoft commented 1 month ago

Thank you @overlookmotel

if there need any help at JavaScript/TypeScript-side or any testing, i would like to help with that

overlookmotel commented 1 month ago

Thanks very much for the offer. I've made a note to call on you when we have type def generation working.

escaton commented 1 month ago

@dalisoft could you please help me understand how Propose 2 can work with dynamic value?

dalisoft commented 1 month ago

@escaton It will work as expected because all values (even dynamic) will be as types so it gets as 1 | 2 for example instead of type number.

escaton commented 1 month ago

In order for TS to infer something with as const assertion, that something needs to be literal, which is impossible for dynamic values. From the TS diagnostic:

A 'const' assertions can only be applied to references to enum members, or string, number, boolean, array, or object literals.
dalisoft commented 1 month ago

@escaton Seems i don’t understand what you means by dynamic values. please give me example of dynamic values and i will try to use as const. i am not pro TS developer, just average developer with TS experience

escaton commented 1 month ago

What happens when AST is parsed by Rust and then transferred to JS is conceptually not different from JSON.parse So how would you use as const to get type inference in this case?

const ast = JSON.parse('{"body": ["ImportDecalration","ExportDecalration"]}')

ast.body
// ^?
dalisoft commented 1 month ago

For such case I don’t know but with for my own project i am did

escaton commented 1 month ago

Indeed, that will work, but this is not quite the mainstream usage of the ast parsers :)

danielroe commented 1 week ago

I've been looking at using the node bindings for oxc-parser (did an experiment to use oxc instead of esbuild + acorn for nuxt). Because of the slight but significant differences from acorn, it would be hugely helpful to have types for the oxc AST. Let me know if there's anything I can do to help, either in implementing the type generation, or in testing the output. 🙏

overlookmotel commented 1 week ago

@danielroe Thanks for the offer. I'm afraid that's going to be a while, because generating types is tied up with other major changes we're making to how AST data is transported from Rust to JS. And for the same reasons, it's unfortunately not something that's really accessible to contributors to work on either.

I'm really sorry that's a rubbish answer! We would dearly like to receive help from the community, but it's just a lot more complicated than it might appear. It goes deep!

When we get movement on this and have type generation working, I hope you don't mind if we take up your offer to help with testing.