Closed MrPrezident closed 4 days ago
Actually no need to do the tree edit. I don't know why I included that in the example. Here is a simpler example showing an incorrect match:
# Initialize test query (match identifier with all caps)
query_text = '((identifier) @test (#match? @test "^[A-Z]$"))'
python_query = python_language.query(query_text)
# Original Source Code
orig_src = b'def myfunc():\n pass\n'
def read_callable_bytes(byte_offset, point):
byte = orig_src[byte_offset:byte_offset + 1]
return byte
# Parse original tree and query
orig_tree = parser.parse(read_callable_bytes)
orig_captures = python_query.captures(orig_tree.root_node)
# Print results
print(f"orig_sexp:{orig_tree.root_node.sexp()}")
print(f"orig_text:{orig_tree.root_node.text}")
print(f"orig_captures:{orig_captures}")
Output:
orig_sexp:(module (function_definition name: (identifier) parameters: (parameters) body: (block (pass_statement))))
orig_text:None
orig_captures:[(<Node type=identifier, start_point=(0, 4), end_point=(0, 10)>, 'test')]
In this case - a read callable does not have the ability to index/remember the source input, so a query match predicate can't execute on that
Is there a fundamental reason why read callable is not used for query? It seems inefficient to have to pass in the entire source code on every keystroke.
Also, can you point me to where the code is for handling the match predicate?
Is there a fundamental reason why read callable is not used for query? It seems inefficient to have to pass in the entire source code on every keystroke.
You cannot get the text at a specific point for the match to execute unlike using a buffer, sure, some read callbacks might be able to, but it's not a guarantee hence why it's omitted
Also, can you point me to where the code is for handling the match predicate?
@amaanq let me know what you think of this PR. I tested this out and it works great for me. I don't see why this wouldn't work for any type of read callback.
@amaanq any love or feedback on this PR?
Hey @MrPrezident I believe upstream (Rust/Wasm) doesn't keep text from sources that use a "read_callable" as well, so I'd like to implement it there first before here - I'm also a bit concerned about the performance cost about keeping the text in certain cases when conditionally switching the input based on external programmatic sources (e.g. a test in Rust will run forever until a timeout is reached - wouldn't that be expensive to keep? https://github.com/tree-sitter/tree-sitter/blob/6019b7c84cd5a7f580d57bb981315afd9afec826/cli/src/tests/parser_test.rs#L676
I believe upstream (Rust/Wasm) doesn't keep text from sources that use a "read_callable" as well, so I'd like to implement it there first before here
Hmm.. ok. I can try to look into that.
I'm also a bit concerned about the performance cost about keeping the text in certain cases when conditionally switching the input based on external programmatic sources
but it's not keeping the text, it's just keeping the callback, or am I missing something?
Well it collects the bytes from the callable right, that's what I meant sorry for not clarifying, like in that timeout test it's possible that it might accumulate a lot of bytes before the timeout is hit.
Maybe a flag in core to enable keeping the source text when using a callable would be nice
Yeah but that test you are talking about is doing parsing, not querying. I didn't make any changes that would affect what happens to the callback during parsing.
The read_callable feature doesn't seem to be working correctly. When I try to use it, the queries get messed up. Here is an example.
Here is the output. The expectation is that there will be no matches on any of the queries. Notice that the new2_captures matched "myfunc" even though it is not capitalized. This only happens when using the read_callable callback.