Error: bad export type for `tree_sitter_typescript_external_scanner_create`: undefined

[PF4Public]

There is a particularly weird issue with tree-sitter that occurs in vscode-l10n if building vscode with electron acting as node (ELECTRON_RUN_AS_NODE=1 electron). Started with electron version 21 onwards.

This could be solved by calling the real node instead of electron, but since it didn't happen in earlier versions of electron, probably there exists another fix for this?

The issue appears as follows:

Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined
Unhandled Rejection at: Promise Promise {
  <rejected> RuntimeError: abort(Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined). Build with -s ASSERTIONS=1 for more info.
      at se (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:6761)
      at k (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:4462)
      at Pe (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:12582)
      at c (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10613)
      at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10879
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:254:10
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:243:22
} reason: RuntimeError: abort(Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined). Build with -s ASSERTIONS=1 for more info.
    at se (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:6761)
    at k (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:4462)
    at Pe (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:12582)
    at c (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10613)
    at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10879
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:254:10
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:243:22

The error tree_sitter_tsx_external_scanner_create comes from https://github.com/microsoft/vscode-l10n/blob/b941cad5b485dad98dc33bfa43b2c0542f8f7e44/l10n-dev/src/ast/analyzer.ts#L41-L52

 static #tsGrammar: Promise<Parser.Language> = (async () => { 
    await initParser; 
    return await Parser.Language.load( 
        path.resolve(__dirname, 'tree-sitter-typescript.wasm') 
    ); 
 })(); 
 static #tsxGrammar: Promise<Parser.Language> = (async () => { 
    await initParser; 
    return await Parser.Language.load( 
        path.resolve(__dirname, 'tree-sitter-tsx.wasm') 
    ); 
 })(); 

Both wasm files come pre-built from @vscode/l10n-dev. Changing the version of @vscode/l10n-dev to 0.0.24 changes the error (although that may be a race condition between the two):

Unhandled Rejection at: Promise Promise {
  <rejected> Error: bad export type for `tree_sitter_typescript_external_scanner_create`: undefined
      at reportUndefinedSymbols (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:19748)
      at postInstantiation (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17027)
      at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17752
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:270:10
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:259:22
} reason: Error: bad export type for `tree_sitter_typescript_external_scanner_create`: undefined
    at reportUndefinedSymbols (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:19748)
    at postInstantiation (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17027)
    at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:17752
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:270:10
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:259:22

Lines 259 and 270 from the error look as follows:

__privateAdd(ScriptAnalyzer, _tsxParser, (async () => {
  await initParser;
  const parser = new import_web_tree_sitter.default();
  parser.setLanguage(await __privateGet(_ScriptAnalyzer, _tsxGrammar));
  return parser;
})());

__privateAdd(ScriptAnalyzer, _tsxGrammar, (async () => {
  await initParser;
  return await import_web_tree_sitter.default.Language.load(
    path.resolve(__dirname, "tree-sitter-tsx.wasm")
  );
})());

In this case tree_sitter_typescript_external_scanner_create is absent from tree-sitter-tsx.wasm, but present in tree-sitter-typescript.wasm.

I have no idea, why is it trying to load a function that is absent from wasm file.

It is interesting that electron-19 and electron-20 show no such behaviour. See fresh logs here.

I have identified that reportUndefinedSymbols comes from emscripten: https://github.com/emscripten-core/emscripten/blob/8874899ad4f345f1880b6876489395844d3d07e1/src/library_dylink.js#L725-L727

        if (!flags.allowUndefined) {
          reportUndefinedSymbols();
        }

But I couldn't find, where does flags.allowUndefined come from. Maybe flag variable got changed between electron 20 and electron 21 such that undefined symbols are no longer allowed? Anyway this seems to go way outside of my knowledge and I may be plain wrong with this conclusion, but I do hope you can enlighten me.

For your reference:

• Mentioned in vscode issue
• vscode-l10n issue

@TylerLeonhardt asked to kindly ping him as well.

[TylerLeonhardt]

I wanted to add another detail... you can easily reproduce this issue by running using Node.js 19 as well:

npx @vscode/l10n-dev export ./folderWithTSFiles

Error: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined
    at reportUndefinedSymbols (/home/codespace/.npm/_npx/68546583dcb2c24b/node_modules/web-tree-sitter/tree-sitter.js:1:19748)
    at postInstantiation (/home/codespace/.npm/_npx/68546583dcb2c24b/node_modules/web-tree-sitter/tree-sitter.js:1:17027)
    at /home/codespace/.npm/_npx/68546583dcb2c24b/node_modules/web-tree-sitter/tree-sitter.js:1:17752
    at async /home/codespace/.npm/_npx/68546583dcb2c24b/node_modules/@vscode/l10n-dev/dist/cli.js:269:10

@deepak1556 gave some advice:

Decompiled the problematic tree-sitter-tsx.wasm file with wasm-decompile and verified that the symbol tree_sitter_typescript_external_scanner_create was defined, however emscripten ends up reporting it as undefined at https://github.com/emscripten-core/emscripten/blob/6f0238367348f5e33605b3f0a42455fb3a2d739f/src/library_dylink.js#L232 , next good step to debug is to recompile the wasm module with assertions and see what went wrong.

It not working in Node 19 as well definitely hints to a v8 breakage. Electron 22 and Node 19 both use V8 10.8.

[SamStenner]

This is still an issue on the latest version of VS Code:

Version: 1.78.2 (Universal)
Commit: b3e4e68a0bc097f0ae7907b217c1119af9e03435
Date: 2023-05-10T14:44:45.204Z
Electron: 22.5.2
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: 10.8.168.25-electron.0
OS: Darwin arm64 22.1.0
Sandboxed: Yes

bad export type for `tree_sitter_typescript_external_scanner_create`: undefined

[hendrikvanantwerpen]

@TylerLeonhardt asked me to have a look at this. From what I can tell, this is not a tree-sitter-typescript specific issue, but instead a tree-sitter problem.

There are several tree-sitter issues that seem related, although none of them mention the exact error. There are open PRs (tree-sitter/node-tree-sitter#127 / tree-sitter/node-tree-sitter#134) that address node version compatibility. Perhaps those would fix this problem?

[TylerLeonhardt]

So we do actually need a fix not only in node-tree-sitter but also web-tree-sitter for this, I think. Going to get an issue going there as well.

[Alpcom]

Any update on this topic ?

[okineadev]

The problem is really in the NodeJS version as said @TylerLeonhardt

To solve the problem, you need to switch to NodeJS version >=18.15.x and <19

You can look at the official documentation, it was written there - https://github.com/microsoft/vscode/wiki/How-to-Contribute#prerequisites