Dynamic injection grammar contributions

[mjbvz]

Problem https://github.com/mjbvz/vscode-lit-html/issues/14

Many users have requested that we colorization the contents of all JavaScript template strings as html. However I think that turning this on by default with no way to disable it would be incorrect. Instead, I want to have a setting that allows users to enable or disable this behavior.

Proposal The current colorization of html tagged templates is provided by an injection grammar. In order to support treating all template strings as html, I would like to be able to contribute this injection grammar dynamically. I can see this working in a few possible ways:

• Add a when clause to contributed grammar injections
• Add a vscode api that allows registering (and possibly unregistering) of injection grammars

The current workaround is to make the extension rewrite its package.json or its contributed grammar (with the changes only taking effect after the next load). I believe that vetur already does something like this

[aeschli]

The current way for users to select a different kind of highlighting is by installing/uninstalling the extension that contributes the grammar/injection. Isn't that good enough for your use case?

The challenge of your proposal is that it involves quite some change

• new API to dynamically add grammars/injections
• extension activation point. (onGrammar)

[aeschli]

@mjbvz The current way for users to select a different kind of highlighting is by installing/uninstalling the extension that contributes the grammar/injection. Isn't that good enough for your use case?

[mjbvz]

Sort of but having to install multiple extensions with the same basic functionality is a poor user experience. My expectation is that I install the lit-html plugin for example, and then configure it for my use case

[mechatroner]

I also need dynamic grammars generation for my "Rainbow CSV" extension. I want to add rainbow highlighting grammars dynamically since there are infinite number of separators (consider multi-character separators), plus for each separator there can be multiple grammars depending on separator escaping policy. Related ticket: https://github.com/mechatroner/vscode_rainbow_csv/issues/1 Currently I am planning to include another 5-10 additional grammars into package.json for most common separator characters, but this won't cover all cases.

[jeff-hykin]

The C++ syntax could use toggle-able syntax options. @aeschli Just allowing users to toggle individual injections would suffice for me (even if it requires a page load).

Here's several more use cases that add up and make separate extensions impractical:

• Highlighting different versions of the same language C++98, C++03, C++11, C++17, C++20
• Adding Library highlighting like the std lib, boost lib, Mac built-ins, and OpenGl lib
• Highlighting embedded assembly code with a particular type of assembly (YASM, AGC, SIC, SNES 65816, GAS, etc)
• There is an existing issue here for highlighting google-style constant names.
• Adding Regex syntax inside of all raw string literals

User's probably won't even realize those are options are available unless they're in the settings somewhere. Creating a separate extension for each one of those is too many repos for a dev to maintain effectively.

What seems to end up happening instead is; an opinionated choice is made and then everyone else has to accept it. For example:

• Python just assumes every raw string == a regex pattern, even though thats not the case
• Python assumes that you're using >3.6 which allows for f"" strings
• JSX is simply included in javascript, even though its not valid javascript and there is even a separate .jsx syntax language
• Python includes # TODO tagging, even though its not a part of the language
• Javascript includes @param tagging, even though its not part of the language
• etc

[n370]

The current way for users to select a different kind of highlighting is by installing/uninstalling the extension that contributes the grammar/injection. Isn't that good enough for your use case?

The challenge of your proposal is that it involves quite some change

* new API to dynamically add grammars/injections

* extension activation point. (onGrammar)

I was fiddling with that yesterday. Created an extension for myself but that yelled the same results as setting editor.tokenColorCustomizations. It seems that something is taking precedence over when the use case is such as described at https://github.com/microsoft/vscode/issues/81020

cc @mjbvz

[mechatroner]

I wonder if this issue should be merged with #68647 ?

[ghost]

:( Literally just needed this feature. I use many versioned languages such as C++ MATLAB Python

I hope you revisit it if and when you get time