Fix token types to use LSP token types

[DavyLandman]

The original implementation was half working, due to an incorrect interpretation of the standard. This commits fixes that. For backwards compatibility: it still supports the legacy ones from rascal and also the wrong text-mate ones.

The rendering of Rascal is the same, except that string and numeric literals are now both highlighted as string (where before they were not highlighted). This is something we can only fix in the rascal grammar.

Also, side-effect, the list of tokens send to the client is less, as we only try and send those that have an category and not all the plain ones.

Fixes #366

TODO:

• [x] extend documentation for DSL
• [ ] (not part of this PR) tune categories in Rascal grammar

[urbanfly]

Will this PR make syntax highlighting Rascal code itself better/different? (like numbers in Rascal code will now be distinct?)

[DavyLandman]

Will this PR make syntax highlighting Rascal code itself better/different? (like numbers in Rascal code will now be distinct?)

Yes, they will now be highlighted (since before they were just the regular text color). But it requires some further tuning to get a difference between the string literals and the numeric literals.

[DavyLandman]

I want to merge this, but the windows runner is crashing on something weird. same case for main branch.

[sonarcloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud