Question about things to backport into VS Code grammar

[dummdidumm]

Hi, I'm one of the maintainer of the Svelte for VS Code extension. Stoked to see that you're pushing to get Svelte syntax highlighting in proper shape on GitHub again!

Looking through the commits I see that you made some tweaks to the existing grammar from the language-tools repo. Could you elaborate a bit on these changes and if it would be worthwhile to backport some or all of them to our VS Code grammar? Particularly interested in the "fix regexes commit".

Thanks!

[umanghome]

Hi, @dummdidumm! Thank you for all your work on the extension! I'm listing down all the issues I ran into below:

1. I wasn't able to add https://github.com/sveltejs/language-tools as a source for grammar directly because linguist picked up the JSON files here and ignored the YAML file. Linguist also said the JSON files were invalid grammar. Linguist also does not support pointing to a specific file, which meant I cannot use the language-tools repo as the source of grammar. Additionally, the answer to a question about the file extensions allowed meant that I had to rename the file extension from .yaml to .YAML-tmLanugage.

2. Next, linguist spit out a bunch of errors when it tried to add grammar from the .YAML-tmLanugage file:

   
   1 errors found in new grammar 'repository `vendor/grammars/svelte-syntax-highlighting-grammar` (from https://github.com/umanghome/svelte-syntax-highlighting-grammar.git)':
   - Grammar conversion failed. File `svelte.YAML-tmLanguage` failed to parse: 58 error(s) decoding:

• 'Repository[attributes-directives].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-directives].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-directives].BeginCaptures[%!s(int=3)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-directives].BeginCaptures[%!s(int=4)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-directives].BeginCaptures[%!s(int=5)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-directives].Patterns[0].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-interpolated].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-keyvalue].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-keyvalue].Patterns[0].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[1].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[1].Captures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[1].Captures[%!s(int=3)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[1].Captures[%!s(int=4)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[3].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[attributes-value].Patterns[3].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[comments].Captures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[comments].Patterns[0].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[comments].Patterns[0].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[comments].Patterns[0].Patterns[0].Captures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[interpolation].Patterns[0].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[interpolation].Patterns[0].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-begin].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-begin].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-begin].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-end].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-end].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-block-end].EndCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-keywords].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-keywords].Captures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[1].Patterns[0].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[2].Patterns[1].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[2].Patterns[1].Patterns[1].Captures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[2].Patterns[1].Patterns[2].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[3].Patterns[0].EndCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-modes].Patterns[4].Patterns[0].Captures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-void].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-void].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[special-tags-void].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-end-node].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-end-node].Captures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-end-node].Captures[%!s(int=3)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-end-node].Captures[%!s(int=4)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-general-end].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-general-end].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-general-end].EndCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-general-start].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-general-start].EndCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-lang-start-attributes].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-lang].BeginCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-lang].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-name].Patterns[0].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-name].Patterns[0].Captures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-name].Patterns[0].Captures[%!s(int=3)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-start-node].Captures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-start-node].Captures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-void].BeginCaptures[%!s(int=1)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-void].BeginCaptures[%!s(int=2)]' expected type 'string', got unconvertible type 'int'
• 'Repository[tags-void].EndCaptures[%!s(int=0)]' expected type 'string', got unconvertible type 'int'

Compilation failed. Aborting

I was able to [convert all the integer object keys to strings](https://github.com/umanghome/svelte-syntax-highlighting-grammar/commit/b835751fea8a32f60eb5894c5ab71669628828ac) and these errors went away.

3. After that, it gave me errors about invalid regexes:

7 errors found in new grammar 'repository vendor/grammars/svelte-syntax-highlighting-grammar (from https://github.com/umanghome/svelte-syntax-highlighting-grammar.git)':

• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=(if|key|then|catch|html).*?)...": lookbehind assertion is not fixed length (at offset 31))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=const.*?)\G": lookbehind assertion is not fixed length (at offset 12))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=each.*?)\G": lookbehind assertion is not fixed length (at offset 11))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=await.*?)\G": lookbehind assertion is not fixed length (at offset 12))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=debug.*?)\G": lookbehind assertion is not fixed length (at offset 12))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=(use|transition|in|out|anima...": lookbehind assertion is not fixed length (at offset 36))
• Invalid regex in grammar: source.svelte (in svelte.YAML-tmLanguage) contains a malformed regex (regex "(?<=(let|class|style):).*$": lookbehind assertion is not fixed length (at offset 22))

Compilation failed. Aborting

I did some searching on the Internet and was able to [update the regexes](https://github.com/umanghome/svelte-syntax-highlighting-grammar/commit/e4ce4681fd8e193897088f4273f55ad27091a66e) based on [a stack overflow answer](https://stackoverflow.com/a/40078049/4176188). The errors went away.

4. Finally, linguist warned about `css.postcss` not being in scope.

1 warnings found when compiling new grammar 'repository vendor/grammars/svelte-syntax-highlighting-grammar (from https://github.com/umanghome/svelte-syntax-highlighting-grammar.git)': OK! added grammar source 'vendor/grammars/svelte-syntax-highlighting-grammar' new scope: source.svelte

• Missing include in grammar: source.svelte (in svelte.YAML-tmLanguage) attempts to include source.css.postcss but the scope cannot be found

These warnings are not fatal, but may mean the syntax highlighting on GitHub.com may not be as expected.

I did [a similar fix back in 2018](https://github.com/umanghome/svelte-atom/commit/adb5935d88775812a641645aed0651d708c4c71d) while adding support to linguist for Svelte. This is because unlike [scss](https://github.com/github/linguist/blob/737d12e24ad5f739934d29216229b8a22336e9ef/lib/linguist/languages.yml#L5907) and [friends](https://github.com/github/linguist/blob/737d12e24ad5f739934d29216229b8a22336e9ef/lib/linguist/languages.yml#L3514), [`postcss` is not in the `css` scope in linguist](https://github.com/github/linguist/blob/737d12e24ad5f739934d29216229b8a22336e9ef/lib/linguist/languages.yml#L5059). [This commit](https://github.com/umanghome/svelte-syntax-highlighting-grammar/commit/141678580a15fd68163a1452929abe0aa5649494) fixed the scope of postcss in the grammar.