Unable to highlight code in raw block

[xiaoxuan-yu]

Describe the bug When try to use tinymist in my project, I found the code in raw block is not highlighted. By inspecting editor token and scope, the results shows only typst.

Package/Software version: VSCode version(Help -> About):

Vesion: 1.90.0
Commit: 89de5a8d4d6205e5b11647eb6a74844ca23d2573
Sate: 2024-06-04T19:43:07.605Z
Electron: 29.4.0
ElectronBuildId: 9593362
Chromium: 122.0.6261.156
Node.js: 20.9.0
V8: 12.2.281.27-electron.0
OS: Linux x64 6.5.0-1023-oem

tinymist extension version: v0.11.0. Get it by tinymist --version in terminal.

tinymist 
Build Timestamp:     2024-06-17T05:47:43.517039979Z
Build Git Describe:  v0.11.11-dirty
Commit SHA:          1c653d5fd28706e27077f8a2d49e6e7f072c2bb2
Commit Date:         None
Commit Branch:       None
Cargo Target Triple: x86_64-unknown-linux-gnu
Typst Version:       0.11.1

Is that a unimplemented feature or a bug?

[Myriad-Dreamin]

This is implemented but introduces bug so I disabled it.

https://github.com/Myriad-Dreamin/tinymist/blob/main/syntaxes/textmate/fenced.ts

The case of bug, when your code in raw block has grammar errors, the highlighting after the raw block will be messy:

The document ```c++
class BuggyCxxCode
``` will be messy.
/// vscode continues parsing c++ grammar after the block
class is hightlighted.

[Enter-tainer]

I wonder how markdown deals with this

[equt]

Would the Embedded languages - Syntax Highlight Guide be helpful?

@Enter-tainer VSCode's markdown syntax is located here and looks like

https://github.com/microsoft/vscode/blob/42b3bac02e37836393b4c4b46fcc91aa03e02aa8/extensions/markdown-basics/syntaxes/markdown.tmLanguage.json#L698-L730

And I see many embedded langauge projects follow the same pattern, say Vue

https://github.com/vuejs/language-tools/blob/e4e8c8ca14dc564bf9043a625dd704b32bdc69d0/extensions/vscode/syntaxes/vue.tmLanguage.json#L90-L121

not sure why the link preview don't work here :(

[Myriad-Dreamin]

The regex tricks from markdown.tmLanguage don't help us, since it cause false highlighting in some cases. Regarding to having semantic highlighting, we would like to make syntax highlighting work only if it is 100% correct. Concludingly, I adapt the rules from vue.tmLanguage and generate all rules to cover cases when the ticks of fence is less or equals to 6.

---

Failure explanation, the following begin/while pattern usually fails:

{
    "begin": "(^|\\G)(\\s*)(.*)",
    "while": "(^|\\G)(?!\\s*([`~]{3,})\\s*$)",
    "contentName": "meta.embedded.block.cpp source.cpp",
    "patterns": [
        {
            "include": "source.cpp"
        }
    ]
}

The above rules don't exactly match fences ([`~]{3,}) (if I read it correctly), sometimes causing messy highlighting.