Textmate grammar does not handle more than one location on a single line

[rodinaarssen]

Describe the bug

The Textmate grammar breaks Rascal's syntax highlighting if more than one location literal with three consecutive slashes appear "consecutively" (i.e., without a location literal with just two consecutive slashes in-between).

To Reproduce

Steps to reproduce the behavior: Note that from the second slash in the second literal

Desktop (please complete the following information):

• Context: VS Code extension
• Rascal Version 0.40.8-SNAPSHOT (~current main)
• Rascal-lsp version: 0.12.0-head (current main)

[sungshik]

Thank you for reporting. Here's what happens:

• For instance, on this line, the TextMate tokenizer assigns the following scopes:

  if (|unknown:///| == l && |unknown:///| == l) {
  //             ^^^^^^^^^^^^^^^^^^^^^            constant.regexp
  //                                  ^^^^^^^^^^^ comment

• The semantic highlighter subsequently tries to overwrite these scopes to meta.embedded to remove the formatting (using our own uncategorized semantic token type):

  

• Issue 1: However, while the overwrite does remove the color (comment is a different shade of gray in this theme), it doesn't remove italic, despite the fontStyle: italic in the token/scope inspector.

• Issue 2: Moreover, VS Code seems to use only scopes assigned by the TextMate tokenizer to determine which brackets are included/excluded to match brackets. Brackets in a comment scope are excluded. As a result, the closing bracket on line 116 in the screenshot is not properly matched to the opening bracket. This is the same "related issue 2" in this comment.

[DavyLandman]

It looks like we should be opening an issue on microsoft/vscode repo about this interaction bteween the textmate & the semantic tokens. Or not @sungshik ? as #431 also gets triggered sometimes, and appears to fall in the same category.

[sungshik]

I agree. I think the matching+highlighting of brackets in comments in the current release of the extension (i.e., before TextMate) is a symptom of the same issue. E.g., the purple brackets of (c) on line 2 in this screenshot:

Do you know if anyone looked into this before? #214 seems related (but not exactly the same).

[DavyLandman]

I've briefly looked into the brackets to see that there didn't seem to be a nice way to control where it shouldn't be active.

[sungshik]

Ok, issue submitted.