How to tell the tree sitter what to color

[jeff-hykin]

Whats the problem?

Because of the tree sitter generating a tree that is (thankfully) different from a TextMate result, there isn't a strong system for mapping colors to tree sitter tags.

Current Implementations/Shortcomings

Currently there are some setting-options that are mapped to hand-picked Tree Sitter tags, which is useful for testing/debugging, but it is a far cry from having full theme support. Atom uses a mapping of tree sitter tags to TextMate scopes and then just uses the old TextMate selector. However, in my opinion, adding the Tree Sitter parser and keeping the TextMate selectors is like taking the time to replace the engine of a rusty car, but not bothering to fix the brakes/tires/interior and paint job. The TextMate selector system is in need of replacement at least as much as the TextMate parser.

Discussion topic

The ideal solution would be a simple api/standard for themes to tell the tree sitter what tree sitter tags they want to color. If VS Code is going to upgrade, then this new system will need to at least be as capable as the old TextMate system. That means there will likely be selectors, like CSS selectors, that allow the theme to describe which parts of the code it wants to color. It will also likely involve creating a naming standard in order to support cross-language theming.

Ideal features would include:

• selecting parent/child scopes (ex: how the Material theme colors nested .json keys)
• being able to overlap adjectives (color tokens that are both keyword AND variable)
• being able to set only a few generic terms and have support for many languages
• being able to limit a selector to a particular language
• being able to select sibling tokens
• being able to negate adjectives (keyword AND NOT "this")

This issue was created to discuss the different features and implementations that would help make theme maker's lives easier.

I'll update this in a few weeks with more details, along with potential methods for making selectors run very efficiently.

[Arcanemagus]

Since it looks like you haven't seen it yet, there's a "Tree-sitter native" list of scopes defined here: tree-sitter/highlight-schema

Atom doesn't currently use it, since all themes are designed around the TextMate scope names, but that is where I would probably start if you don't want to translate to the TextMate scopes.

[jeff-hykin]

Thanks! That is going to be helpful, and I didn't know about it

[jeff-hykin]

Major headway was made on this here. Maxbrunsfeld, who has basically written all of code for the tree sitter, released plans for TSQuery, a way to select nodes in the parsed representation of the tree sitter. This is the exact kind of foundation needed to resolve this issue.