Open xloc opened 1 month ago
Interesting, issue doesn't show up when using the C library or Rust bindings:
[lillis@LaptopWill] ~/projects/tree-sit-large-file-test (main) ⚡
❯ gcc -I /home/lillis/projects/tree-sitter/lib/include test.c /home/lillis/projects/tree-sitter-json/src/parser.c /home/lillis/projects/tree-sitter/libtree-sitter.a -o test
[lillis@LaptopWill] ~/projects/tree-sit-large-file-test (main) ⚡
❯ ./test
{
"h6aTZoqB1C": "IppZxiu3b4NPay7bTT9f2pB9oHrttFEEjneB6CGjvy4zh2u6FdOJUAYqAaS39KWkfrG7PlbOrHrH8laiJCg%
[lillis@LaptopWill] ~/projects/tree-sit-large-file-test (main) ⚡
❯ cargo run
Compiling tree-sit-large-file-test v0.1.0 (/home/lillis/projects/tree-sit-large-file-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.62s
Running `target/debug/tree-sit-large-file-test`
"{\n\"h6aTZoqB1C\": \"IppZxiu3b4NPay7bTT9f2pB9oHrttFEEjneB6CGjvy4zh2u6FdOJUAYqAaS39KWkfrG7PlbOrHrH8laiJCg"%
Thanks a lot for the quick reply!
So the problem is very likely from the nodejs side
I hit the same problem in tree-sitter-javascript 0.21.4, however setting an appropriate buffer size solves the problem.
// Define the options with bufferSize
const options: Parser.Options = {
bufferSize: 1024 * 1024, // Set the bufferSize to 1 MB (1024 KB)
};
parser.parse(jsCode, undefined, options);
Awesome! I didn’t realized that there is an option for buffer size. Will investigate further.
Thanks a lot!
Problem
Not sure if it is the correct place to post this issue, but ...
I'm using the node binding of tree-sitter, and
Parser.parse(string)
reports the following error when I try to parse a large i18n json file (~1.2M).Code
I tried to use the
Parser.Input
callback instead of passing in a string, butnode.text
doesn't load properlythis outputs
For details, please refer to my repo of minimal reproducible example.
https://github.com/xloc/tree-sit-large-file-test
Thanks in advance!
Steps to reproduce
Expected behavior
the file can be parsed correctly without errors
Tree-sitter version (tree-sitter --version)
tree-sitter=0.21.1; tree-sitter-json=0.21.0
Operating system/version
Windows 10.0.19044 N/A Build 19044; Node.js v22.4.0