Failing tests with latest tree-sitter versions

[susliko]

At the moment CI piplines use the tree-sitter fork, which is at version 0.20.0 Running tests on main branch via latest versions (0.20.1 and 0.20.2 tested) of tree-sitter results in a few failing tests: Steps to reproduce:

cargo install tree-sitter-cli
tree-sitter generate
tree-sitter test

[ahelwer]

Huh, wonder how this wasn't caught by CI. I noticed this on my mac a couple weeks ago but thought it was just a local thing (I develop locally on Windows). Will look into it. Are you reproducing this locally on a mac? Or are you saying that using the non-forked tree-sitter causes those test failures? Because that's to be expected - the fork has some fixes for get_column to count code points instead of bytes, which is why it's failing on unicode tests.

[susliko]

I've tested multiple combinations locally on Linux:	Generate by	Test by	Test result
fork	fork	ok
fork	upstream	error
upstream	fork	~error~ ok
upstream	upstream	error

So, even with the fork-generated grammar latest tree-sitter versions fail to parse some specs with disj_list and conj_list correctly

[ahelwer]

The more surprising thing here is that grammars generated with upstream fail on fork. The get_column changes are present in the runtime tree-sitter C library, not the generated code. I will do the following:

• Rebase fork changes on latest tree-sitter release and create another fork release
• Actually get around to writing good tests for the get_column changes so I can merge them with upstream

[susliko]

he more surprising thing here is that grammars generated with upstream fail on fork

Ah, that's my copy-paste fault. Sorry for that

[susliko]

Neovim ts playground also shows the behavior described above. Syntax highlighting is broken as well

[ahelwer]

If you would believe it, I had to hack together a fork of neovim using my tree-sitter fork for the demo lol: https://github.com/tlaplus-community/neovim

Upstream neovim uses upstream tree-sitter, so get_column counts bytes instead of codepoints.

Ref https://github.com/tree-sitter/tree-sitter/issues/1405

[ahelwer]

Required changes have been merged into upstream tree-sitter and a new tree-sitter release was issued yesterday. Now we are just waiting for the tree-sitter-cli NPM module to be updated; see https://github.com/tree-sitter/tree-sitter/issues/1613

Hopefully not too long after that the neovim tree-sitter library dependency will be updated, and we'll get proper unicode column position counting in neovim itself.

[clason]

Neovim (nightly) already has tree-sitter 0.20.3 since yesterday ;)

[maxbrunsfeld]

0.20.4 is out on npm.