No highlighting with typst-lsp.semanticTokens set to disable

[Enivex]

• Component:
  • VSCode Extension:
• Extension version: 0.9.3
• LSP version: 0.9.3
• OS version and name: Windows 11 23H2

• I am on the latest stable version of the extension/LSP.
• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

If I set "typst-lsp.semanticTokens": "disable" in my settings.json, then I get no syntax highlighting at all. I was expecting it to fall back to the rudimentary highlighting that existed before.

[nvarner]

That was my original intention, but VS Code seems to make this hard. Semantic tokens are enabled/disabled as part of the LSP, so can be toggled controlled by configuration. However, VS Code TextMate syntax highlighting (what was used before) is controlled by package.json, so it's either enabled or disabled unless you modify and reinstall the extension.

If TextMate and semantic token highlighting are both enabled at once, the semantic tokens appear on top of TextMate (this is the intention behind semantic tokens; the stated purpose is to highlight just a few things based on semantics that TextMate can't do by syntax, like based on if a variable is mutable or not). However, the TextMate grammar is out of date; I don't think it's been updated since it was pulled from an early version of Typst (but see #256 for a possible small update). It's hard to maintain the TextMate grammar because Typst's syntax is still in flux, whereas semantic token highlighting "just works" because it's based on Typst's own parser. If the TextMate grammar makes up the background of the semantic tokens, errors in it could highlight the code wrong, and it would have some cost for VS Code to perform TextMate highlighting.

If there is some way of enabling or disabling highlighting by configuration, I'd love to incorporate it. Without that, the best solution would probably be to make a separate extension that just provides highlighting.