bug: incremental_selection at ending space reports E5108

[xudyang1]

Did you check existing issues?

• [x] I have read all the tree-sitter docs if it relates to using the parser
• [x] I have searched the existing issues

Tree-Sitter CLI Version, if relevant (output of tree-sitter --version)

No response

Describe the bug

When cursor is at the end of a line with a space, calling incremental selection in markdown file throws E5108.

Steps To Reproduce/Bad Parse Tree

1. nvim repro.md
2. type ahello world, with an extra space at the end
3. exit insert mode, move cursor to the ending space and use keymap to call node_incremental
4. Error message is shown:

E5108: Error executing lua: ...ashfs-root/usr/share/nvim/runtime/lua/vim/treesitter.lua:167: attempt to index local 'node_or_range' (a n
il value)
stack traceback:
        ...ashfs-root/usr/share/nvim/runtime/lua/vim/treesitter.lua:167: in function 'get_node_range'
        ...im/lazy/nvim-treesitter/lua/nvim-treesitter/ts_utils.lua:279: in function 'update_selection'
        ...treesitter/lua/nvim-treesitter/incremental_selection.lua:19: in function <...treesitter/lua/nvim-treesitter/incremental_selec
tion.lua:15>

Expected Behavior/Parse Tree

Parse the node correctly

Repro

No response

[xudyang1]

Here is what I tried in neovim:

Adding the following lines in nvim-treesitter/lua/nvim-treesitter/ts_utils.lua, line 207:

vim.print("ts.is_in_node_range(root, line, col):", ts.is_in_node_range(root, line, col))
vim.print("root:range(): ", root:range())
vim.print("range: {line, col, line, col+1}", { line, col, line, col + 1 })

1. repro.md with "hello " (one extra space, error occurs)

   ts.is_in_node_range(root, line, col):
   false
   root:range():
   0
   0
   0
   5
   range: {line, col, line, col+1}
   { 0, 5, 0, 6 }
   E5108: Error executing lua: ...ashfs-root/usr/share/nvim/runtime/lua/vim/treesitter.lua:167: attempt to index local 'node_or_range' (a n
   il value)
   stack traceback:
       ...ashfs-root/usr/share/nvim/runtime/lua/vim/treesitter.lua:167: in function 'get_node_range'
       ...im/lazy/nvim-treesitter/lua/nvim-treesitter/ts_utils.lua:283: in function 'update_selection'
       ...treesitter/lua/nvim-treesitter/incremental_selection.lua:19: in function <...treesitter/lua/nvim-treesitter/incremental_selec
   tion.lua:15>

2. repro.md with "hello " (two extra spaces, works)

   root:
   <userdata 1>
   ts.is_in_node_range(root, line, col):
   true
   root:range():
   0
   0
   1
   0
   range: {line, col, line, col+1}
   { 0, 6, 0, 7 }

3. repro.py (or repro.any_file_type) with "hello " (one extra space, works)

   root:
   <userdata 1>
   ts.is_in_node_range(root, line, col):
   true
   root:range():
   0
   0
   1
   0
   range: {line, col, line, col+1}
   { 0, 5, 0, 6 }

So, I guess tree-sitter-markdown produces a wrong root:range() when cursor is at the extra ending space (should be 0 0 1 0 instead of 0 0 0 5).

[MDeiml]

I would guess that this is most likely a neovim specific issue. In that case it would be best to report it at https://github.com/nvim-treesitter/nvim-treesitter.

[xudyang1]

Closed, reported to neovim

[xudyang1]

Ending space is not included in $.inline https://github.com/neovim/neovim/issues/31015#issuecomment-2450416154

[MDeiml]

I still don't think the parser is at fault here. It produces a syntax tree without throwing an error, and node_incremental as I understand should work irrespective of the semantic meaning of that syntax tree.

I can test if changing the inline node like you suggest fixes the issue though.

[MDeiml]

I added a new branch, where the final space should now be contained in the inline node, could you check if this helps? https://github.com/tree-sitter-grammars/tree-sitter-markdown/tree/paragraph_inline_with_space

[xudyang1]

I added a new branch, where the final space should now be contained in the inline node, could you check if this helps? https://github.com/tree-sitter-grammars/tree-sitter-markdown/tree/paragraph_inline_with_space

The paragraph_inline_with_space branch works. I also replace markdown.so and markdown_inline.so in neovim's parser directory, running nvim --clean repro.md also works as expected.

(document [0, 0] - [1, 0]
  (section [0, 0] - [1, 0]
    (paragraph [0, 0] - [1, 0]
      (inline [0, 0] - [1, 0]))))

I tested it with tree-sitter test in tree-sitter-markdown subdirectory, and many examples failed with:

                  (inline                                            // unexpected
                    (block_continuation))))                // unexpected
                  (inline)                                           // expected
                  (block_continuation)))                   // expected

I believe those are false positive as the paragraph rule was changed.

Thank you for your hard work on it. Closing this.