Support multiline and overlapping semantic tokens

[michaelpj]

There are client capabilities for if the client supports these. But if they don't, we can still support them gracefully I think. The conversion functions can take the client capabilities and then implement a workaround if the capabilities aren't supported:

• For overlapping tokens, split the overlapped token so into the part before the overlapping token and the part after the overlapping token, so you get something like: [<string>"hello ${a} world"][<variable>a] -> [<string>"hello ${][<variable>a][<string>} world"]
• For multiline tokens, split them at the line breaks into one token per line.

[soulomoon]

There are client capabilities for if the client supports these. But if they don't, we can still support them gracefully I think. The conversion functions can take the client capabilities and then implement a workaround if the capabilities aren't supported:

• For overlapping tokens, split the overlapped token so into the part before the overlapping token and the part after the overlapping token, so you get something like: [<string>"hello ${a} world"][<variable>a] -> [<string>"hello ${][<variable>a][<string>} world"]

• For multiline tokens, split them at the line breaks into one token per line.

This idea look pretty nice.

[soulomoon]

I wonder how overlappingToken look like in SemanticTokenRelative. we might have nagative delta, but UInt is unsigned. 🤔

[michaelpj]

It's also quite possible that it doesn't actually work out. It's not implemented in vscode, so it could just not make sense.