Experimental Tree Sitter investigation

[jasonwilliams]

Hey @jeff-hykin

I was looking into tree sitter for VSCode. Do you know if my thinking was on the right track?

My understanding is that for things to not break there would need to be some mapping back to textmate scopes in order for things further up the chain to still work? Is that what this extension does?

[jeff-hykin]

That would be direction with the least amount of work, and that's how Atom did it. It did end up making it complicated for atom.

While there are line-by-line parts, the parsing is more complicated in practice.

I believe it goes a bit more like this

• do lines independently (for immediate/wrong colors that are better than nothing)
• then parse the whole doc
• save the parsing data in an incremental way
• then when there's an edit, update the line and what comes after it

The tree sitter isn't line based though so updating the current-line-and-below wouldn't work. I believe it has an incremental update system, but it would need hooks for knowing what changed.

Once those details have been worked out then it would be theoretically possible to generate tokens using the tree sitter. There would need to be a function that accepts a tree sitter abstract syntax tree as an argument, and return a bunch of TextMate tokens

[jeff-hykin]

But doing that also ruins half of the benefits of the tree sitter for me personally. TextMate tokens are problematic

[jasonwilliams]

Thanks @jeff-hykin do you know why it currently parses line by line? It seems if tree-sitter was added in you could replace the loop and just pass the whole lot into tree sitter and get all the tokens back at once instead?

I believe it has an incremental update system, but it would need hooks for knowing what changed.

I don’t think that’s an issue I believe there are onChange hooks for that, you would just need to wire it up

But doing that also ruins half of the benefits of the tree sitter for me personally. TextMate tokens are problematic

yeah you would need to start somewhere, I wonder if once that’s done you could start replacing other parts that work with textmate scopes.

[jeff-hykin]

do you know why it currently parses line by line?

it's a very sad "performance" hack. For example if you took a JavaScript or C++ program and remove all the new lines, it becomes impossible for the editor to parse it with TextMate (for any program of a normal size).

 It seems if tree-sitter was added in you could replace the loop and just pass the whole lot into tree sitter and get all the tokens back at once instead?

yeah probably haha. It is more than an order of magnitude faster than Textmate. The tree sitter might also have caching that would make re-parsing the same thing much faster.

I don’t think that’s an issue I believe there are onChange hooks for that, you would just need to wire it up

I agree. There are dectorator hooks and such but they're not close to being performant.

I wonder if once that’s done you could start replacing other parts that work with textmate scopes

The hardest part are the themes that other people have written. Sublime themes, Atom themes, Textmate(the editor) themes were all ported to VS Code. It's a hard chicken and egg problem. Getting the tree sitter in there though would be a good start