Support multiple `SemanticTokensProvider` per document selector

[BoykoAlex]

Currently there is very basic support for multiple semantic tokens provider for the same language. The first provider that computed some tokens is used. (See #135580). This feature was also requested in #135599 but didn't look like anything got pushed to fix it...

However, in reality, tokens from multiple semantic token providers need to be taken into account. For example Java annotations in the case of Spring can have JPQL, HQL or native SQL queries depending on annotation parameters and JARs on the classpath. Seems that best approach for such use case in Semantic Tokens. However, since JDT LS also has SemanticTokensProvider any Spring specific SemanticTokenProvider would conflict with the JDT LS provider.

Looking at https://github.com/microsoft/vscode/blob/main/src/vs/editor/common/services/semanticTokensProviderStyling.ts#L142 i have a feeling that the restriction of effectively having one token provider giving tokens can be lifted if switched to SemanticTokens[]

(cc: @martinlippert)

[VSCodeTriageBot]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[VSCodeTriageBot]

:slightly_smiling_face: This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[BoykoAlex]

I have noticed that a few semantic token providers for the same language where only one provides non-null results isn't working so not sure if this https://github.com/microsoft/vscode/pull/135602 works as expected or maybe I misunderstood something...

I took semantic-tokens-sample from https://github.com/BoykoAlex/vscode-extension-samples. Added:

In the activate(...) function of the extension.ts

context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider({ language: "java" }, new JavaSemanticTokensProvider(), legend));

Where the JavaSemanticTokensProvider:

import {CancellationToken, DocumentSemanticTokensProvider, ProviderResult, SemanticTokens, TextDocument} from "vscode";

export class JavaSemanticTokensProvider implements DocumentSemanticTokensProvider {
    provideDocumentSemanticTokens(document: TextDocument, token: CancellationToken): ProviderResult<SemanticTokens> {
        return Promise.resolve(null);
    }
}

Also changed activation on event in package.json:

    "activationEvents": [
        "onStartupFinished"
    ],

After all this i see that JDT LS provided semantic tokens are not present.

(See https://github.com/BoykoAlex/vscode-extension-samples/tree/multi-semantic-token-providers and don't forget to install Java extension pack or RedHat Java extension)