semanticTokensProvider is missing Connection implementation

[LordOfDragons]

https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide indicates support for language servers to provide semantic token information. The node language server has the ServerCapabilities.semanticTokensProvider information you can define. But there is no Connection protocol implementation (like for example Connection.onDocumentSymbol). This makes it impossible to provide semantic tokens.

Is node language server missing this implementation or is it hidden somewhere?

[dbaeumer]

It gor scoped under languages. Look for connection.languages.semanticTokens

[LordOfDragons]

I see. But the interface has a problem. You need to use SemanticTokenBuilder to build the return value of this callback function. This class has the following function:

push(line: number, char: number, length: number, tokenType: number, tokenModifiers: number): void

It is totally unclear how this is supposed to work.

For tokenType I assume the number there is the index of the type in the array used in the server options. That would make sense. But for the tokenModifiers this does not make sense. There can be multiple modifiers but only one number can be specified. How is this number build? Here documentation is missing.

[LordOfDragons]

Also in the documentation (https://vscode-api.js.org/classes/vscode.SemanticTokensBuilder.html) the constructor takes the legend. The actual implementation available through NPM has a 0-argument constructor only.

[dbaeumer]

That is the VS Code API which is different than the LSP API since LSP works with other editors than VS Code as well. The corresponding LSP code is here: https://insiders.vscode.dev/github/microsoft/vscode-languageserver-node/blob/release/voiceless-hedgehog-salmon/server/src/common/semanticTokens.ts#L127

[LordOfDragons]

I have no access to this link.

Also what I said about 0-argument constructor is against the LSP api SemanticsTokensBuilder:

import { SemanticTokenModifiers, SemanticTokenTypes, SemanticTokensBuilder, SemanticTokensLegend, integer } from "vscode-languageserver"

I've based the implementation on this page: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide . According to this page the following packages are included: On the client side:

"engines": {
    "node": "*"
},
"dependencies": {
    "chevrotain": "^10.4.2",
    "copyfiles": "^2.4.1",
    "rimraf": "^3.0.2",
    "vscode-languageserver": "^7.0.0",
    "vscode-languageserver-textdocument": "^1.0.8"
},

On the server side:

"engines": {
    "vscode": "^1.63.0"
}
"dependencies": {
    "@vscode/vsce": "^2.24.0",
    "minimatch": "^9.0.3",
    "npm": "^10.8.0",
    "vscode-uri": "^3.0.8"
}

[dbaeumer]

Sorry here are correct link: https://github.com/microsoft/vscode-languageserver-node/blob/main/server/src/common/semanticTokens.ts#L127

The difference in the args come from the fact that for an LSP server the legend is part of the initialize request. This is different compared to using a token builder directly in the extension host API since there no such hand shake has taken place.

[LordOfDragons]

Okay. But this still doesn't answer the question on what value the "tokenType: number" and "tokenModifiers: number" require. I've got a a token type (the ones provided by LSP are strings) and list of token modifiers (the ones provided by LSP are string). How are these mapped to the respective arguments of the push function call?

Hence as a concrete example, how would this mapping look like?

• tokenType = SemanticTokenTypes.class
• tokenModifiers = [SemanticTokenModifiers.declaration, SemanticTokenModifiers.abstract, SemanticTokenModifiers.deprecated]

[dbaeumer]

The legend is agreed on in the initialize handshake. See https://github.com/microsoft/vscode-languageserver-node/blob/main/testbed/server/src/server.ts#L120 as an example.

After that you can use the agreed numbers to pass them to the semantic token builder. Here is an example that assign random tokens and modifiers: https://github.com/microsoft/vscode-languageserver-node/blob/main/testbed/server/src/server.ts#L654

[LordOfDragons]

Just so I understand this correctly if the mapping is for example like this:

• SemanticTokenTypes.class => 2
• SemanticTokenModifiers.declaration => 1
• SemanticTokenModifiers.abstract => 3
• SemanticTokenModifiers.deprecated => 6

then the call would be like this?

push(line, char, length, 2, (1<<1) + (1<<3) + (1<<6))

[dbaeumer]

From the specification

Token types are looked up by index, so a tokenType value of 1 means tokenTypes[1]. Since a token type can have n modifiers, multiple token modifiers can be set by using bit flags, so a tokenModifier value of 3 is first viewed as binary 0b00000011, which means [tokenModifiers[0], tokenModifiers[1]] because bits 0 and 1 are set.

[LordOfDragons]

I see. I got it working now. Would be nice to have this information in a public space where it can be found more easily.

[dbaeumer]

The specification is public here: https://microsoft.github.io/language-server-protocol/