Slow speed

[sainttttt]

I was wondering if the slow speed when loading a nim file is going to be addressed? I checked the previous issues and it seems like there's a rewrite going on - is it still in progress? And will it improve the parsing speed? Thanks!

[alaviss]

a rewrite going on - is it still in progress?

It's already done before 0.1.0 :p

I was wondering if the slow speed when loading a nim file is going to be addressed?

It will as soon as I figure out what is the real issue.

I assume that you're talking about neovim, then one of the issues is that the highlight queries used there is too big: It took 460ms for neovim to load the queries, according to nvim-treesitter tests.

An another issue seems to be that it just takes long for the parser binary to be loaded, which might be due to how big the parser itself is. Reducing the generated parser size is still a major unresolved challenge, so improvements on that front is unlikely at the moment.

[sainttttt]

Gotcha thanks! Yeah I am talking about neovim. So is work being done on speeding up the highlight queries? (Apart from reducing the generated parser size)

[alaviss]

So is work being done on speeding up the highlight queries?

The highlight queries in nvim-treesitter are not maintained by me, and while I do intend to improve it, my focus is on other projects at the moment.

[clason]

Yes, the Nim queries in nvim-treesitter are complete overkill, and a simplification would be very welcome. (Such hyper-detailed queries belong to a special-purpose plugin or personal config.)

Regarding parser size (which definitely is an issue), https://github.com/nvim-treesitter/nvim-treesitter/wiki/Parser-Development has some helpful links (especially the tree-sitter wiki and the Github blog).

@alaviss I take the opportunity of drawing your attention to a recent breaking change in tree-sitter that will make tree-sitter-nim fail to generate on the next release (0.22.3); see https://github.com/tree-sitter/tree-sitter/pull/2577#issuecomment-2183969412 and the following. It looks like the grammar will have to be adapted.

[alaviss]

Regarding parser size (which definitely is an issue), https://github.com/nvim-treesitter/nvim-treesitter/wiki/Parser-Development has some helpful links (especially the tree-sitter wiki and the Github blog).

Yea, I've reviewed most of the resources there in the (many) rewrites of this grammar. At this point I'm pretty much out of ideas on how to simplify it further.

Many of the issues I have right now can be traced to how termination of pretty much every complex statements are dependent on context-sensitive sets of tokens (eg. have to go to external scanner), which seems to hinder TS state merging. I've pulled some compression techniques (_inhibit* tokens which compresses else, elif, except, finally and the )]}.} set) to cut the parser size by half (it was >100MB before), but some often seen tokens like , cannot be compressed like so.

@alaviss I take the opportunity of drawing your attention to a recent breaking change in tree-sitter that will make tree-sitter-nim fail to generate on the next release (0.22.3); see tree-sitter/tree-sitter#2577 (comment) and the following. It looks like the grammar will have to be adapted.

That's a very weird breakage to have... I'll follow up on that PR to see where we can go from here, this was needed to implement an exception in Nim's operator parsing rule where *: has to be parsed as * and :.

EDIT: I already fixed this particular one, PR coming soon.

[clason]

Yep, can confirm that works! Thanks!

[aMOPel]

Maintainer of the current nvim-treesitter nim queries here.

I'm not planning to write and test a sleaker version of the queries.

Feel free to contribute.