Add semantic highlighting data structures, encoding and default capabilities

[martin-azpillaga]

To facilitate the implementation of semantic highlighting in language servers, I suggest adding the following reusable parts to this plugin:

1. Data structures to hold the default list of semantic token categories, semantic token modifiers and semantic tokens:

   
   data class SemanticToken(val position: Position, val type: SemanticTokenType, val modifiers: List<SemanticTokenModifier>)

enum class SemanticTokenType (val legendName: String) { NAMESPACE("namespace"), CLASS("class"), ENUM("enum"), INTERFACE("interface"), STRUCT("struct"), TYPE_PARAMETER("typeParameter"), TYPE("type"), PARAMETER("parameter"), VARIABLE("variable"), PROPERTY("property"), ENUM_MEMBER("enumMember"), DECORATOR("decorator"), EVENT("event"), FUNCTION("function"), METHOD("method"), MACRO("macro"), LABEL("label"), COMMENT("comment"), STRING("string"), KEYWORD("keyword"), NUMBER("number"), REGULAR_EXPRESSION("regexp"), OPERATOR("operator") }

enum class SemanticTokenModifier(val legendName: String, val bit: Int) { DECLARATION("declaration", 1), DEFINITION("definition", 2), READ_ONLY("readonly", 4), STATIC("static", 8), DEPRECATED("deprecated", 16), ABSTRACT("abstract", 32), ASYNCHRONOUS("async", 64), MODIFICATION("modification", 128), DOCUMENTATION("documentation", 256), DEFAULT_LIBRARY("defaultLibrary", 512) }

2. A function to encode a list of semantic tokens to the numerical encoding the LSP uses:
``` kotlin
fun encode(tokens: List<SemanticToken>): SemanticTokens {
    var lastLine = 1 // ? To offset that Kolasu lines start at 1
    var lastColumn = 0
    val data = mutableListOf<Int>()
    for (token in tokens) {
        data.addAll(listOf(
            token.position.start.line - lastLine,
            if (token.position.start.line == lastLine) token.position.start.column - lastColumn else token.position.start.column,
            token.position.end.column - token.position.start.column, // ! assumes tokens are in a single line
            token.type.ordinal,
            token.modifiers.sumOf { it.bit }
        ))
        lastLine = token.position.start.line
        lastColumn = token.position.start.column
    }
    return SemanticTokens(data)
}

3. Configuration to enable semantic highlighting capabilities in the server initialization request:

   capabilities.semanticTokensProvider = SemanticTokensWithRegistrationOptions().apply {
   legend = SemanticTokensLegend(SemanticTokenType.values().map { it.legendName }, SemanticTokenModifier.values().map { it.legendName });
   full = Either.forLeft(true)
   }

[loradd]

🆙

I am fine with this suggestion, perhaps we could merge it before the article on syntax/semantic highlighting comes out so that we do not need to say that the feature is still not public.

What do you think?

[martin-azpillaga]

I think we could also extract this bit of logic to make the APIs a bit friendlier:

override fun semanticTokensFull(params: SemanticTokensParams): CompletableFuture<SemanticTokens> {
    val ast = files[params.textDocument.uri]?.root ?: return CompletableFuture.completedFuture(SemanticTokens())
    val tokens = semanticTokens(ast)
    return CompletableFuture.completedFuture(encode(tokens))
}

open fun semanticTokens(ast: T): List<SemanticToken> = listOf()

Like this, developers just need to populate the list of semantic tokens starting from an ast, which I find more intuitive than dealing with LSP specific constructs.

Let's discuss whether to publish these changes before the article quickly in the next technical meeting.