Semantic Token Handler not getting invoked

[jamal-ahmad]

Following the example language server under sample/SampleServer, i've created a sematic token provider for my lanaguage. However tokenize and other functions are never getting invoked. I'm confused about what the issue is? Am i missing some configs/settings?

I'm using:

• OmniSharp.Extensions.LanguageServer v-0.18.1
• vscode-languageclient v-^6.1.3
• dotnet v-3.1.102

Main entry point for language server:

using System;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using OmniSharp.Extensions.LanguageServer.Protocol.Window;
using OmniSharp.Extensions.LanguageServer.Server;
using Serilog;

namespace server
{
    class Program
    {
        private static void Main(string[] args) => MainAsync(args).Wait();
        private static async Task MainAsync(string[] args)
        {
            Log.Logger = new LoggerConfiguration()
                        .Enrich.FromLogContext()
                        .WriteTo.File("log.txt", rollingInterval: RollingInterval.Day)
                        .MinimumLevel.Verbose()
                        .CreateLogger();
            Log.Logger.Information("This only goes file...");

            var server = await LanguageServer.From(options =>
                options
                    .WithInput(Console.OpenStandardInput())
                    .WithOutput(Console.OpenStandardOutput())
                    .ConfigureLogging(
                        x => x
                            .AddSerilog(Log.Logger)
                            .AddLanguageProtocolLogging()
                            .SetMinimumLevel(LogLevel.Debug)
                    )
                    .WithHandler<TextDocumentHandler>()
                    .WithHandler<SemanticTokensHandler>()
                    .WithServices(x => x.AddLogging(b => b.SetMinimumLevel(LogLevel.Trace)))
                    .WithServices(services =>
                    {
                        services.AddSingleton(provider => {
                            return new BufferManager(CreateLogger<BufferManager>(provider));
                        });
                    })
                    .OnInitialize(async (server, request, token) =>
                    {
                        server.Window.LogInfo("executing pre-initialize steps");
                    })
                    .OnInitialized(async (server, request, response, token) =>
                    {
                        server.Window.LogInfo("executing post-initialize steps");
                    }).OnStarted(async (languageServer, token) =>
                    {
                        languageServer.Window.LogInfo("server started");
                    })
            );
            await server.WaitForExit;
        }

        private static ILogger<T> CreateLogger<T>(IServiceProvider provider) {
            var loggerFactory = provider.GetService<ILoggerFactory>();
            return loggerFactory.CreateLogger<T>();
        }
    }
}

Semantic Token Handler (BufferManager below is a very simple class to store text document contents):

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using grammer;
using Microsoft.Extensions.Logging;
using OmniSharp.Extensions.LanguageServer.Protocol;
using OmniSharp.Extensions.LanguageServer.Protocol.Client.Capabilities;
using OmniSharp.Extensions.LanguageServer.Protocol.Document.Proposals;
using OmniSharp.Extensions.LanguageServer.Protocol.Models;
using OmniSharp.Extensions.LanguageServer.Protocol.Models.Proposals;

namespace server
{
#pragma warning disable 618
    using TokenInfo = Tuple<SemanticTokenType, IList<SemanticTokenModifier>>;
    class SemanticTokensHandler : SemanticTokensHandlerBase
    {
        private readonly ILogger _logger;
        private readonly BufferManager _bufferManager;

        public SemanticTokensHandler(
            ILogger<SemanticTokensHandler> logger,
            BufferManager bufferManager
        ) : base(
            new SemanticTokensRegistrationOptions
            {
                DocumentSelector = DocumentSelector.ForLanguage("pepper"),
                Legend = new SemanticTokensLegend(),
                Full = new SemanticTokensCapabilityRequestFull
                {
                    Delta = true
                },
                Range = true
            }
        )
        {
            _logger = logger;
            _bufferManager = bufferManager;
            _logger.LogInformation("SemanticTokensHandler created");
        }

        public override async Task<SemanticTokens> Handle(
            SemanticTokensParams request,
            CancellationToken cancellationToken
        )
        {
            _logger.LogInformation($"SemanticTokensParams called for ${request.TextDocument.Uri.GetFileSystemPath()}");
            var result = await base.Handle(request, cancellationToken);
            return result;
        }

        public override async Task<SemanticTokens> Handle(
            SemanticTokensRangeParams request,
            CancellationToken cancellationToken
        )
        {
            _logger.LogInformation($"SemanticTokensRangeParams called for ${request.TextDocument.Uri.GetFileSystemPath()}");
            var result = await base.Handle(request, cancellationToken);
            return result;
        }

        public override async Task<SemanticTokensFullOrDelta?> Handle(
            SemanticTokensDeltaParams request,
            CancellationToken cancellationToken
        )
        {
            _logger.LogInformation($"SemanticTokensDeltaParams called for ${request.TextDocument.Uri.GetFileSystemPath()}");
            var result = await base.Handle(request, cancellationToken);
            return result;
        }

        protected override async Task Tokenize(
            SemanticTokensBuilder builder,
            ITextDocumentIdentifierParams identifier,
            CancellationToken cancellationToken
        )
        {
            var content = _bufferManager.GetContent(identifier.TextDocument.Uri);
            await Task.Yield();

            if (content != null)
            {
                var (lexer, tokens) = GrammerUtils.getAllTokens(content);
                _logger.LogInformation($"Discovered {tokens.Count} tokens");
                foreach (var t in tokens)
                {
                    var info = _tokenMapping.GetValueOrDefault(t.TokenSource.SourceName, null);
                    builder.Push(
                        t.Line,
                        t.Column,
                        t.StopIndex - t.StartIndex + 1,
                        (info != null) ? info.Item1 : null,
                        (info != null) ? info.Item1 : null
                    );
                }
            }
        }

        protected override Task<SemanticTokensDocument> GetSemanticTokensDocument(
            ITextDocumentIdentifierParams @params,
            CancellationToken cancellationToken
        )
        {
            return Task.FromResult(new SemanticTokensDocument(GetRegistrationOptions().Legend));
        }

        private static TokenInfo TokenInfo(SemanticTokenType type, params SemanticTokenModifier[] modifiers)
        {
            return Tuple.Create<SemanticTokenType, IList<SemanticTokenModifier>>(
                type,
                modifiers
            );
        }

        private static Dictionary<string, TokenInfo> _tokenMapping = new Dictionary<string, TokenInfo>(){
            /* commenting out mapping of tokens */
        };
    }
#pragma warning restore 618
}

On the client side i've set "enableProposedApi": true in the package.json and the entry points look like so:

import * as path from 'path';
import { ExtensionContext, workspace } from 'vscode';
import { LanguageClient, LanguageClientOptions, ServerOptions, Trace, TransportKind } from 'vscode-languageclient';

function generateClient() {
    let serverExe = 'dotnet';
    let runDllPath = path.join(__dirname, '/path/to/Server.dll');
    let debugDllPath = path.join(__dirname, path.join(
        "..",
        "..",
        "server",
        "bin",
        "Debug",
        "netcoreapp3.1",
        "server.dll"
    ));
    let serverOptions: ServerOptions = {
        run: { command: serverExe, transport: TransportKind.pipe, args: [runDllPath] },
        debug: { command: serverExe, transport: TransportKind.pipe, args: [debugDllPath] }
    };

    let clientOptions: LanguageClientOptions = {
        documentSelector: [
            {
                pattern: '**/*.pkd',
            }
        ],
        synchronize: {
            configurationSection: 'MyLangaugeServer',
            fileEvents: workspace.createFileSystemWatcher('**/*.pkd')
        },
    };

    let client = new LanguageClient(
        'MyLangaugeServerClient',
        'MyLanguage Client',
        serverOptions,
        clientOptions
    );
    client.trace = Trace.Verbose;
    return client;
}

export function activate(context: ExtensionContext) {
    // Create the language client and start the client.
    const client = generateClient();
    let disposable = client.start();
    context.subscriptions.push(disposable);
}

// this method is called when your extension is deactivated
export function deactivate() { }

[david-driscoll]

Semantic highlight only works with the latest language client (v7 betas). The latest beta added a breaking change as well, the next version will address that however.

[jamal-ahmad]

ah! is that that case only for vscode language client or for all LSP client impls e.g. would atom work?

[david-driscoll]

It depends on the client, I haven't paid attention to atom for a while so I'm honestly not sure (however I doubt it). The nice thing is that even if a given client doesn't support something, if they end up adding it in the future, then it will just automatically light up. 😄

[Pyromuffin]

I'm running into the same issue with the latest vscode and the latest commit of this repo on master. @david-driscoll mentioned that it only works with language client v7 beta? Where can I find this? I'm not sure I see anything called v7 on the vscode repo.

I'm currently reviving a semantic highlighting server, and I am surprised to find that it doesn't work at all anymore and the tokenize callback isn't getting called. It would be very helpful to me to have a set of known working libraries/clients so I can get back to testing my server.

Thanks!

[jamal-ahmad]

I haven't been able to make V7 work yet either.

NPM has the V7 versions of the client but they're all beta. Look under "versions"

vscode-languageclient 7.0.0-next.12

npm i vscode-languageclient@7.0.0-next.12

Or

yarn add vscode-languageclient@7.0.0-next.12

Should get you the latest beta

[Pyromuffin]

I was able to get the sample server to work with semantic tokens by manually changing the dependences in vscode-testextension/packages.json to

"vscode-languageclient": "^7.0.0-next.12", "vscode-languageserver-protocol": "^3.16.0-next.10"

after that the test extension works with semantic tokens!

[jamal-ahmad]

wait so you modified the package.json of a a dependency inside node_modules?

[david-driscoll]

If you're using the next.12 version of the vscode languageclient make sure you're using v0.18.2 as that has the fix to make it work for that. If you're using next.10 or lower the last version that supports that version of the spec is v0.18.1. Keep in mind the spec has been a proposal all this time and it can / will change the wind blows. This is the reason the main types are annotated with the Obsolete attribute.

[jamal-ahmad]

Seems like they've changed how Language clients are created as well? LanagugeClient is no longer exported from the vscode-languageclient (see output at the bottom). I do see CommonLanguageClient which is kinda similar but it doesn't accept server options. @david-driscoll can you please point me to where you're tracking the proposed beta changes?

src/extension.ts:3:10 - error TS2305: Module '"../node_modules/vscode-languageclient/lib/common/api"' has no exported member 'LanguageClient'.

3 import { LanguageClient, LanguageClientOptions, Trace } from 'vscode-languageclient';
           ~~~~~~~~~~~~~~

Found 1 error.

error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[david-driscoll]

Yeah they've added support for browser and node language clients!

changing your import to import { LanguageClient, LanguageClientOptions, Trace } from 'vscode-languageclient/node'; should solve your problem.

The next version of the protocol spec is here: https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/ I also try to follow the repo that contains the vscode client/server https://github.com/Microsoft/vscode-languageserver-node

[jamal-ahmad]

Confirmed that the semantic highlighting works with the import that @david-driscoll suggested. Thanks for the help @david-driscoll 🙂 Current versions that i'm using:

//client/package.json
"dependencies": {
    "vscode-languageclient": "7.0.0-next.12"
},
"devDependencies": {
    "@types/glob": "^7.1.3",
    "@types/mocha": "^8.0.0",
    "@types/node": "^12.11.7",
    "@types/vscode": "^1.50.0",
    "@typescript-eslint/eslint-plugin": "^4.1.1",
    "@typescript-eslint/parser": "^4.1.1",
    "eslint": "^7.9.0",
    "glob": "^7.1.6",
    "mocha": "^8.1.3",
    "typescript": "^4.0.2",
    "vscode-test": "^1.4.0"
}

<ItemGroup>
    <PackageReference Include="Antlr4.Runtime.Standard" Version="4.8.0" />
    <PackageReference Include="GuiLabs.Language.Xml" Version="1.2.46" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="3.1.3" />
    <PackageReference Include="OmniSharp.Extensions.LanguageServer" Version="0.18.2" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.0.1" />
    <PackageReference Include="Serilog.Sinks.Debug" Version="1.0.1" />
    <PackageReference Include="Serilog.Sinks.File" Version="4.1.0" />
</ItemGroup>

[david-driscoll]

👍🏼