Improve support for Neovim 0.9 LSP Semantic Tokens

jrwrigh did a great job adding initial support for Neovim's LSP Semantic Tokens in https://github.com/sainnhe/sonokai/pull/86 (backported to Everforest in 1ba66ff71cd3e8c369399d793cba2ca2e7eaa11b).

Since LSP Semantic Tokens offer an even more granular view on the code tokens than Tree-sitter does, there is still a bit of work ahead of us to apply correct highlight semantics uniformly (as described in palette.md).

I will use this issue as an umbrella for identifying potential gaps.

Note Not all of the gaps identified below are immediately visible to the user. For example, the color scheme currently links both "type" and "built-in type" to the same color, but still uses different groups for them. We should preserve that distinction whenever possible to facilitate customization and potential future adjustments to this color scheme.

(The Go language and the gopls language server will be my only references for the time being.)

Struct fields

In Tree-sitter, struct fields are identified as @field (blue). gopls returns the semantic tokens below, where variable (fg) is a fallback for the more specific/accurate variable definition token type.

  - @lsp.type.variable.go links to Fg priority: 125
  - @lsp.mod.definition.go links to @lsp priority: 126
  - @lsp.typemod.variable.definition.go links to @lsp priority: 127

Constants

In Tree-sitter, constants are identified as @constant (fg). gopls returns the semantic tokens below, where variable (fg) is a fallback for the more specific/accurate variable readonly token type.

  - @lsp.type.variable.go links to Fg priority: 125
  - @lsp.mod.definition.go links to @lsp priority: 126
  - @lsp.mod.readonly.go links to @lsp priority: 126
  - @lsp.typemod.variable.definition.go links to @lsp priority: 127
  - @lsp.typemod.variable.readonly.go links to @lsp priority: 127

Method parameters

In Tree-sitter, method and function parameters are identified as @parameter (fg). gopls returns the semantic tokens below, where parameter (fg) is a fallback for the more specific/accurate parameter definition token type.

  - @lsp.type.parameter.go links to Fg priority: 125
  - @lsp.mod.definition.go links to @lsp priority: 126
  - @lsp.typemod.parameter.definition.go links to @lsp priority: 127

Built-in constants

In Tree-sitter, built-in constants such as nil are identified as @constant.builtin (purple italic). gopls returns the semantic tokens below, where variable (fg) is a fallback for the more specific/accurate variable readonly token type.

  - @lsp.type.variable.go links to Fg priority: 125
  - @lsp.mod.defaultLibrary.go links to @lsp priority: 126
  - @lsp.mod.readonly.go links to @lsp priority: 126
  - @lsp.typemod.variable.defaultLibrary.go links to @lsp priority: 127
  - @lsp.typemod.variable.readonly.go links to @lsp priority: 127

Built-in types

In Tree-sitter, built-in types are identified as @type.builtin (yellow italic). gopls returns the semantic tokens below, where type (yellow italic) is a fallback for the more specific/accurate type defaultLibrary token type.

  - @lsp.type.type.go links to YellowItalic priority: 125
  - @lsp.mod.defaultLibrary.go links to @lsp priority: 126
  - @lsp.typemod.type.defaultLibrary.go links to @lsp priority: 127

Namespace

  - @lsp.type.namespace.go links to YellowItalic priority: 125

ℹ️ Go specific

Namespace is technically linked correctly to namespace (yellow italic), but the result feels wrong and overly yellow in a language like Go where everything is explicitly namespaced. The problem was already addressed in #81, so this is a regression.

I decided not to address most of these items for the following reasons:

Struct fields

@lsp.typemod.variable.definition is really for all variable definitions, not only struct fields. There is no token that represents struct fields specifically in the current (3.17) LSP specification. In my opinion, gopls should be using @lsp.type.property here instead.

If we set @lsp.typemod.variable.definition to some foreground color other than fg (e.g. blue or aqua), every variable definition will stand out with that color. On one hand this looks agressive on-screen, on the other hand Tree-sitter doesn't have highlight groups for variable definitions (that I know of) so I see no good reason to do that with LSP semantic tokens either.

Users who want to highlight variable definitions are free to do so, either globally with @lsp.typemod.variable.definition, or per-language with e.g. @lsp.typemod.variable.definition.go, @lsp.typemod.variable.definition.rust, etc.

Note: @lsp.typemod.variable.definition and @lsp.typemod.variable.declaration seem to be used interchangeably by different LSP servers.

Built-in constants

Highlighting @lsp.typemod.variable.defaultLibrary globally turns out to be problematic in certain languages. In Lua, for example, global modules such as table or vim (in Neovim) are treated as such:

I suggest we do it on a per-language basis instead.

Everything else

In the current version of Neovim (0.9.0), highlights for higher level types/typemods take precedence over certain language specific customizations.

For example, given the following priorities for Go's nil:

the highlight for @lsp.typemod.variable.readonly, if set (even without the .go prefix), takes precedence over a language-specific highlight @lsp.typemod.variable.defaultLibrary.go . In my opinion, this is a barrier to per-language configuration.

Again, I suggest that we do things on a per-language basis.

sainnhe / everforest