Tree queries

[omentic]

Hi aMOPel, I've been working on some highlighting queries based off of this repository's grammar. They're currently in development over at https://github.com/helix-editor/helix/pull/6123: once they're done I can PR them to this repo, neovim, and anywhere else that uses tree-sitter.

[aMOPel]

Hi :) I honestly am pretty stuck on the last few syntax nodes. Unless somebody is gonna help me with that grammar, who has a better understanding of tree sitter I probably won't finish it.

Your efforts are probably better spent on

https://github.com/alaviss/tree-sitter-nim/tree/rewrite

[omentic]

I saw that, I'll probably rewrite the queries for it once alaviss finishes (originally I wrote them for alaviss's first grammar). Yours is already quite comprehensive though, it looked like objects are the big thing missing?

Did you manage to find any good documentation for tree-sitter when you were working on the grammar? I've been writing the queries by reading through other existing queries and grammars and sort of guessing how they should be written, for lack of anything better to base them off of: which hasn't worked terribly well. I've been having a particularly hard time with keywords: they have to line up with the grammar.js file somehow and I haven't quite figured out how.

[aMOPel]

it looked like objects are the big thing missing?

I've made issues describing the missing features. One big thing missing are the complex expressions, ie if-expressions etc. And another thing are the various parentheses. There is this construct called "par" in the grammar spec which works like this:

let a = (
  echo "hi"
  5
)
echo a
# prints 
# hi
# 5

But there are also the arbitrary parentheses around expressions like

let a = (1+1)*2

It's a pain to disambiguate those from tuples and function calls, but probably doable. The complex expressions are the big thing I got stuck on though.

Also another big issue is that the grammar.js is so big and complex now, that the resulting parser is absolutely enormous in size. What's pushed right now is 15MB large. When I added more things it grew to 28MB or something.

Did you manage to find any good documentation for tree-sitter when you were working on the grammar?

Can't say so. I haven't even written any queries myself. Your best bet is probably to take inspiration from preexisting code. The nvim-treesitter repo has a huge collection of queries you could probably learn a lot from those.

[omentic]

TIL about par, I don't think I've ever seen that in practice.

By any chance, do you know of a good way to throw this grammar at an example file to generate syntax trees like in tests/corpus? I think neovim has some support for it on their devel branch but it'd be nice if I could just use the grammar directly.

[omentic]

Also, how do you build parser.c?

[aMOPel]

do you know of a good way to throw this grammar at an example file to generate syntax

https://tree-sitter.github.io/tree-sitter/creating-parsers#command-parse

This will parse the file and generate the AST.

I don't know if this helps you but

https://nim-lang.github.io/Nim/macros.html#treeRepr%2CNimNode

This can give you the AST from parser of the Nim compiler

If your questions was about good example files to parse:

https://github.com/nim-lang/Nim/blob/devel/compiler/parser.nim

How about this one?

Also, how do you build parser.c

https://tree-sitter.github.io/tree-sitter/creating-parsers#command-generate

[omentic]

Thanks, I got tree-sitter dumping trees, which is pretty helpful.

On further testing the highlighting doesn't work as well as I thought it did: there's a lot of edge cases, incorrect tree representations, and occasional errors (mostly around type declarations). It's still better than no highlighting though...

[aMOPel]

If it's just about highlights in neovim, I am using this plugin atm:

https://github.com/alaviss/nim.nvim

[aMOPel]

If you care enough you could make an issue listing the failing edge cases, and I could have a look at it

[aMOPel]

I actually fixed some issues with the type descriptions etc.

If you still find something, please report it.

I tested it on various large files from the nim repo and it did everything except

1. complex expressions (like if expressions, case expressions etc.) and
2. par (parentheses to group things like - (a + b)).

Of course I expect there to be various bugs still.

[omentic]

I'll open issues for any new problems I find :+1:

[aMOPel]

Btw I have written some highlight queries that comply with the nvim tree sitter interface. Gonna push later

[aMOPel]

pushed on a new branch https://github.com/aMOPel/tree-sitter-nim/blob/nvim_treesitter_queries/queries/highlights.scm

[omentic]

Oh shoot, I wish I saw this earlier. I rewrote my Helix queries to work with object and concept types and generally be less jank (currently pushed at https://github.com/helix-editor/helix/pull/6123).

I have some known issues with my queries:

• Modules like std/[one, two] will get highlighted as variables (while std/one is fine)
• tuple[one, two: string] and tuple[one: string, two: string] are highlighted incorrectly (seq[seq[int]] is fine
• I couldn't figure out a way to distinguish between variables and types in case statements (if there isn't one, probably should do it by case)

[omentic]

Mmm, note to self, helix and nvim-treesitter queries have reversed precedence (later entries override in nvim, vice versa for helix)