microsoft / vscode-textmate

A library that helps tokenize text using Text Mate grammars.
MIT License
562 stars 108 forks source link

JSON key `"id"` breaks the tokenizer #229

Open RedCMD opened 4 months ago

RedCMD commented 4 months ago

The internal tokenizer uses the key "id" as a way to keep track of rules within a grammar However, the parser does not strip "id" keys from the grammar before parsing if a invalid "id" key is hit during tokenizing, it will lead to unpredictable tokenizing and the grammar partially failing

Luckily it seems no one has placed one in the wrong spot (having one at root is fine) tho I assume that's because they would have noticed their grammar no longer working when it is present https://github.com/search?q=path%3A*.tmLanguage.json+%22%5C%22id%5C%22%3A%22&type=code

{
    "scopeName": "source.lang",
    "patterns": [
        {
            "begin": "\"",
            "end": "\"",
            "name": "string",
            "patterns": [ { "include": "#backslash" } ]
        }
    ],
    "repository": {
        "backslash": {
            "match": "\\\\.",
            "name": "constant.character.escape",
            "id": "JSON key \"id\" breaks this rule and its parent"
        }
    }
}