search

zed-industries / zed

Code at the speed of thought – Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
https://zed.dev
Other
49.38k stars 3k forks source link

Cannot build Tree-sitter grammar with external scanner #14901

Open fabjan opened 3 months ago

fabjan commented 3 months ago

Check for existing issues

Describe the bug / provide steps to reproduce it

I'm trying to make an extension for the Standard ML language, using this Tree-sitter grammar that someone has already made: https://github.com/stonebuddha/tree-sitter-sml

But something is not right in the way the parser gets compiled. When loading the extension Zed cannot instantiate the wasm module because the external scanner symbols are not available.

Steps to reproduce

  1. Install my work-in-progress extension: https://github.com/fabjan/zed-sml
  2. Open a Standard ML code file
  3. Open the Zed log
  4. Observe error message about invalid import

Notes

I am not very familiar with Tree-sitter, I may be misunderstanding something. A large language model I asked about this is telling me I need to fix the export.

This is what wasm-objdump says about the compiled module:

❯ wasm-objdump -x -j export ~/Library/Application\ Support/Zed/extensions/installed/sml/grammars/standard_ml.wasm       

standard_ml.wasm:   file format wasm 0x1

Section Details:

Export[3]:
 - func[1] <__wasm_apply_data_relocs> -> "__wasm_apply_data_relocs"
 - func[2] <_initialize> -> "_initialize"
 - func[3] <tree_sitter_standard_ml> -> "tree_sitter_standard_ml"

Is something missing in my configuration, or does the compile_grammar function need to know more about what to export? https://github.com/zed-industries/zed/blob/cd9b25d82728d2c8f2537db3ceccc5e211e8146e/crates/extension/src/extension_builder.rs#L211

Environment

Zed: v0.144.4 (Zed) OS: macOS 14.5.0 Memory: 8 GiB Architecture: x86_64

If applicable, add mockups / screenshots to help explain present your vision of the feature

No response

If applicable, attach your Zed.log file to this issue.

Zed.log

2024-07-21T08:07:34.540732Z [INFO] checking out standard_ml parser
2024-07-21T08:07:34.653749Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:34.752599Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:35.21431Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:35.316168Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:35.317004Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:35.396599Z [INFO] compiling standard_ml parser
2024-07-21T08:07:35.416715Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:35.417312Z [INFO] building git repository, `.git` path in the worktree: "grammars/standard_ml/.git"
2024-07-21T08:07:37.80874Z [INFO] finished compiling extension /Users/fabian/Library/Application Support/Zed/extensions/installed/sml
2024-07-21T08:07:38.047032Z [WARN] Theme "Blanche light" is using a deprecated style property: scrollbar_thumb.background. Use `scrollbar.thumb.background` instead.
2024-07-21T08:07:38.055324Z [INFO] rebuilt extension index in 42.929001ms
2024-07-21T08:07:38.06041Z [INFO] extensions updated. loading 0, reloading 1, unloading 0
2024-07-21T08:07:38.063843Z [INFO] Initializing default prettier with plugins {}
2024-07-21T08:07:38.11348Z [ERROR] failed to load language Standard ML:
Failed to instantiate wasm module: invalid import 'tree_sitter_standard_ml_external_scanner_create'

2024-07-21T08:07:38.327224Z [INFO] Language server with id 0 sent unhandled notification LogMessage:
{
  "level": 1,
  "message": "[INFO] [ghostText] [2024-07-21T08:07:38.323Z] Cancelled during debounce",
  "metadataStr": "[INFO] [ghostText] [2024-07-21T08:07:38.323Z]",
  "extra": [
    "Cancelled during debounce"
  ]
}
noau commented 3 months ago

I've been suffering from similar problems and if you still have this problem, my solution is that the tree-sitter grammar I'm using is using an external C++ scanner, and the C++ API is said to be deprecated, so I switched it to C, and boom, all things runs perfectly. Hope this helps :)

fwcd commented 3 months ago

Porting the scanner to C seems to have fixed the issue for me, thanks!

Also worth noting that the Tree-sitter API now features tree_sitter/array.h as a canonical implementation of a generic dynamic array, which was not too hard to adapt from the previously std::vector-based implementation of the scanner I ported.