vscode-cue: Sublime Text / TextMate / Sourcegraph syntax highlighting support

[slimsag]

I thought cue-lang/vscode-cue#12 was all we would need to get the grammar in syntaxes/cue.tmLanugage working for Sublime Text/TextMate/Sourcegraph, but unfortunately not: I've ran into another instance where the grammar here is not valid.

There are several instances where the grammar specifies a patterns array, without the begin and end keys:

invalid instances

https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L57-L63 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L90-L102 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L132-L137 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L178-L203 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L204-L209 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L225-L235 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L236-L243 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L244-L259 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L264-L313 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L314-L322 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L339-L350 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L367-L372 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L373-L388 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L389-L412 https://github.com/cue-sh/vscode-cue/blob/4dff3d1e9ea19953de460950255885806100b7f2/internal/cmd/gen-syntax/defs.go#L473-L475

I noticed this because Sublime Text 4 fails to validate the grammar prior to conversion to a .sublime-syntax file, this is relatively easy to test but I won't enumerate the steps here unless you're interested.

Both the TextMate website and Sublime website are fairly clear:

• The root patterns array may contain rules
• A rule may contain a nested patterns array, only if it is a begin..end element.

From https://docs.sublimetext.io/reference/syntaxdefs_legacy.html#patterns-array

From https://macromates.com/manual/en/language_grammars#rule_keys

Unfortunately, it seems VS Code has differed here in their tmLanguage implementation and allowed a more lax interpretation - so the grammar here works in VS Code, but not elsewhere.

Sublime is capable of doing a best-effort translation, but it will drop highlighting for all of the mentioned definitions above (which is most of the grammar.) As for what to replace these patterns-only definitions with.. that's a bit trickier.

[myitcv]

If only we had a schema to validate these grammars! ❤️

FWIW, this constraint could be described using a CUE schema with the help of a couple of upcoming proposals (https://github.com/cue-lang/cue/issues/943 and https://github.com/cue-lang/cue/issues/575)

As for the fix, perhaps we do this as part of cue-lang/cue#3425

cc @betawaffle for thoughts on any of this.

[betawaffle]

This was done intentionally in the original code I wrote (if I recall correctly) to avoid duplication where possible.

The fix for this bug is to take every instance of these and replace them with their contents. For example, everywhere #attribute_element is included, you'd instead include #attribute_label, #attribute_nested, and #attribute_string. You'd have to do that recursively obviously, since (for example) #attribute_string has that problem too.

The best way in my mind would be to implement it as a post-processing step, since duplicating all of that by hand sounds really painful, and would make it harder to maintain in the future.

I you implement the whole thing in CUE, there might be a simpler way to do the above, but I'm not certain.