LSP Semantic Tokens support

[Tachi107]

Hi all, do you have any plans to support LSP Semantic Tokens in the default Vim color schemes? Vim itself does not support them, but third party plugins and NeoVim do, so it'd be nice to have semantic colors working without having to install third party themes.

A few (hopefully) useful links:

• https://github.com/prabirshrestha/vim-lsp/pull/1275
• https://github.com/yegappan/lsp/issues/126
• https://github.com/neovim/neovim/pull/21100
• https://github.com/catppuccin/nvim/issues/367
• https://github.com/sainnhe/gruvbox-material/pull/162

[romainl]

Hi.

This project primarily functions as a clearinghouse for color schemes distributed with Vim. As such, the only features we support are those provided by Vim itself: Vim doesn't support "LSP Semantic Tokens" natively so Vim color schemes don't support "LSP Semantic Tokens" natively.

We will, of course, add support for any new highlight group the day it finds its way into :help highlight-groups but I don't see that coming any time soon for anything related to LSP or TreeSitter.

In the meantime, plugins that support those tokens can simply follow the established best practice of linking their custom highlight groups to default ones.

[Tachi107]

Hi, thanks for the reply :)

I'm fairly new to Vim so I wasn't sure if supporting Semantic Tokens was a reasonable request / in scope, but knowing that plugin authors can link additional highlight groups addressed my concerns.

Thanks again!

[atgote]

In the meantime, plugins that support those tokens can simply follow the established best practice of linking their custom highlight groups to default ones.

Will you please share a link/any thoughts on what these "established best practices" are since trying to link existing/default highlight groups to LSP Semantic Tokens I was feeling like reinventing the bicycle.

[romainl]

So… let's imagine for a moment a world without linking.

I work with a new language called FooBar that is not officially supported by Vim so I go ahead and create a custom syntax script with a whole lot of custom highlight groups like FooBarToken, etc.:

syn keyword FooBarToken qfdgqs sqdfsgd gfsqdg fsdgsqfgf dsghq gdfqs

Since the goal is to have those words displayed in color, I need to set a number of attributes for FooBarToken in my own colorscheme:

hi FooBarToken cterm=NONE ctermbg=NONE ctermfg=darkmagenta

So far, so good: I can edit FooBar files with syntax highlighting.

I am a generous guy and I want every other FooBar developers to have the same refined experience, which means that I am going to share my lovely syntax script with the world. And that is where things get way out of hand.

The thousands of colorscheme authors in the wild don't know about FooBar so I will have to ask ALL OF THEM to include ALL my FooBar* highlight groups and, in most cases, the actual color to use will have to be discussed for each highlight group. That is an unspeakable amount of work for me, who has a vested interest in FooBar, but also for those colorscheme authors who, in general, don't particularly care about my favorite language. Imagine the kind of dedication needed to support your colorscheme. This is not socially possible.

Moreover, this would mean that EVERY colorscheme would have to include EVERY highlight group for EVERY existing language and plugin, which would make them HUGE and SLOW. This is not technically possible.

This is a nightmare.

Now… let's get back to reality.

The solution to this problem is for me to "link" all those language/plugin-specific FooBar* highlight groups to a managed number of default highlight groups. With that mechanism, I can do the following in my syntax script:

hi def link FooBarToken Keyword

and be confident that my FooBar tokens will be highlighted appropriately by ANY colorscheme…

• without me having to do an astronomical quantity of work,
• without putting any burden on the shoulder of colorscheme authors,
• without breaking any colorscheme.

That is the best practice I mentioned.

"LSP Semantic Tokens", whether they are available at the first-party level (in Neovim, which we don't support) or at the third-party level (in vim-lsp, for example) are no different in this respect: the maintainer of whatever is defining those highlight groups MUST link them to the default groups, just like everyone else, without putting the whole world in motion.

The maintainer of vim-lsp did just that in the PR linked by OP and anyone interested in adding "LSP Semantic Tokens" support to their plugin is expected to do the same.

[atgote]

The maintainer of vim-lsp did just that in the PR linked by OP and anyone interested in adding "LSP Semantic Tokens" support to their plugin is expected to do the same.

Oh, thanks for this really complete explanation! The main problem for me was that many LSP Semantic Tokens types can be easily linked to default highlight groups but some highlight groups just do not exist in vim like:

parameter
macro
type
namespace
label

so I've created this map:

parameter => key
macro => key
type => class
namespace => key
label => key

but will review how that's done in the mentioned PR again.

Thanks!

[romainl]

You can already use:

• Macro (linked to default group PreProc)
• Type (a default group)
• Label (linked to default group Statement)

For parameter and namespace, you could take a look at how they or semantically similar groups are linked in existing syntax scripts.