Universal Language Injections

[felixfbecker]

Language injections is a feature of IDEs like Webstorm and PHPStorm that auto-detects when a language is use inside another language and then shows syntax highlighting for that injected language. VS Code currently supports this in a few places:

• <style> tags in HTML get CSS highlighting and IntelliSense
• <script> tags in HTML get JS highlighting and IntelliSense
• HTML in a PHP file outside of <?php ?> gets highlighting but no IntelliSense

But there are many other contexts where this would be helpful, that are currently not supported and impossible to all hardcode:

• style attribute in HTML contain CSS
• String literals in languages like PHP often contain SQL, see #1361. PHPStorm supports this for PHP, but Webstorm for example does not support this for NodeJS. It should really be supported in any language.
• String literals commonly contain HTML in Javascript or PHP (e.g. with jQuery, Angular or Vue templates)
• String literals commonly contain GraphQL queries in JavaScript
• HTML event attributes contain Javascript, like onclick="event.preventDefault()"
• Countless HTML attributes in the Angular framework have Javascript-like expressions in them, like ng-if="ctrl.items.length > 0 && someOtherValue", but also the mustache syntax {{ ... }}.
• <script type="text/ng-template"> contain Angular templates which is nothing but HTML. This pattern of templates in script tags exists in many of the major client side MVC frameworks.
• Javascript has RegExp literals. In other languages like PHP, RegExps are passed as strings. This is also the case in tasks.json problem matchers for VS Code (RegExp inside JSON string)

All of these examples currently don't get colored and receive no autocompletion. It would be awesome if VS Code was able to automatically detect languages in any context (like string literals, HTML attributes and tags) by some language patterns, like < > tags for HTML/XML, keywords like SELECT, INSERT, etc. for SQL or template string tags like gql for GraphQL. This could be contributed by extensions or language servers.

This would be one of the best features VS Code could offer.

[bpasero]

@egamma @dbaeumer @aeschli fyi

[felixfbecker]

Added RegExp example

[farfromunique]

Piggybacking: Autocomplete for the injected language would be awesome! It's really disengaging to be typing HTML for output from PHP and have PHP syntax suggestions.

[anyong]

Would love to have SQL syntax highlighting throughout my projects. Currently I always write my SQL in a separate file with syntax highlighting, then copy it into my project where it needs to go. In VS Code it's just highlighted as a normal string, so very hard to catch errors and whatnot.

[ricardobeat]

Javascript and CSS highlighting inside <script> and <style> tags should be a basic feature.

[ciel]

I love this idea. Lack of template coloring is one of the main reasons I kept using WebStorm over Visual Studio Code for some projects.

[MarkPieszak]

I don't know if this would help in VSCodes case, but GregOnNet came up with a brilliant solution for adding HTML highlighting to the back-tick section here for Atom-typescript

https://github.com/TypeStrong/atom-typescript/pull/948

Hoping that might help potentially add this ability to VSCode!

[felixfbecker]

@mahmoudymy I don't think is a good solution to hardcore template support for every framework out there. I want syntax highlighting in all kinds of template strings, in my gulp tasks, in my SQL queries, ...

[polina-c]

How about Mason: perl inside html?

[mikeal]

would love this! :)

[MattMcFarland]

Please add this! I have been working on a choo framework project and because it uses es6 template tags for html and then even the tags inside can be escaped it just turns into a mess with syntax highlighting. I think I cant find anything that works other than jetbrains language injection. Do they have a patent on that?

[MarkPieszak]

@MattMcFarland Atom handles es6 template strings really well at the moment as well, you could use that in the meantime until it's added to Code.

[MattMcFarland]

@MarkPieszak Yes I've tried atom, and it doesnt handle them well enough, any javascript code that is injected for onclick events for example atom thinks is a string. Also FWIW you can use atom grammar in vscode and get similar results.

[a-s-o]

Hello everyone,

I was just wondering whether a plugin can accomplish this. My pitch is to have a language specific plugin that does the injections based on its own settings. For instance, a plugin for js could perform injection based on tagged string templates with the following user configurable settings:

Tag         |  Language
-------------------------------
html        | HTML
css         | CSS
graphql     | GraphQL

Usage would be as follows:

import html from 't7';
import css from 'csjs';
import graphql from 'graphql-tag';

export const TodoComponent = {
  template: ({ styles }) => html`
    <div class=${styles.todoApp}>
      <Header>
        <NewTodo class=${styles.newTodo} />
      </Header>
      <ul>
        ${Todos}
      </ul>
    </div>
  `,

  styles: css`
    .todoApp {
      background: yellow;
    }

    .newTodo {
      display: flex;
    }
  `,

  queries: graphql`
    query getTodos ($user: String) {
      todoList (user: $user) {
        id
        title
        completed
      }
    }
  `
};

So basically, users configure the tag they want to match and are then are require to properly name those tags in scope to identify the syntax to be injected.

Is this theoretically possible in a vscode extension? Or would this require editor level changes? I don't have experience writing a plugin yet, but I am willing to take a stab if possible.

[adamvoss]

I think language servers would need to participate in this to indicate what language a section of a document is. If the complexity can be controlled, this has potential to be very powerful.

For example some programming languages support Markdown in their commenting system. If they could mark comment blocks as markdown then any other extensions for Markdown could be active there.

Another non-HTML example would be allowing YAML extensions to be active in Jekyll front matter.

[felixfbecker]

Yes, a language server knows exactly where (and sometimes what) languages are injected. It could send a notification announcing these ranges, like string literals, comments, script tags, style tags. VS Code can then do embedded syntax highlighting and open these ranges as virtual documents in language servers.

[loganintech]

This could also be very useful for HTML Color in angularjs's template: tag within components. I very much want this 👍

[wdhwg001]

Any updates? I believe this will be one of the most convenient feature for frontend developers. We have HTML, Pug, Vue and a lots of template languages that contains other languages like CSS(SASS, SCSS, LESS, Stylus) and JavaScript(TS, JSX, CoffeeScript). If VSCode support language injection/embedding, those extension writers will no longer need to write CSS/JS highlighting & auto-completing again and again.

[tradeli]

I think the discussions in #670 are related to this.

[mhubig]

This seems to be related: https://github.com/wholroyd/vscode-jinja/issues/19

[noinkling]

Just want to add: you could also utilize comment hints as a fallback for when smart-detection doesn't work or isn't possible. In fact that might be a good first step for implementing such functionality generically. e.g:

db.query(
  // #lang:sql
  `SELECT * FROM ... ;`
)

[tradeli]

There are related discussions going on in #585 , #2915 , #670 , #130 and most recently in #36396 . So, what's your input on this proposal (#1751 by @felixfbecker ) @bmewburn and @aeschli ?

Is it just me, or the docs for Language extensions authoring is lacking some tips for when multiple languages are at use?

[bmewburn]

I think this proposal is a great idea. Though for me the question became: how to get PHP and html working today using the existing protocol and API?

There's not really much documentation for multiple language server setups. It seems kind of experimental and left up to extension authors. The html language server source and history is a good place to see how it can be done.

[rowanG077]

Is this at all on the roadmap? Because this feature would be really usefull.

[robhaswell]

Hi team, are there any updates on this? Is it on the roadmap?

[bmewburn]

Could a new ErrorCode be added?

1. VSCode first tries the server listening on the document languageId.
2. Receives an LanguageRangeError response with info on the embedded languageId and ranges.
3. Creates a virtual doc and whitespace replace undesired ranges.
4. Forwards request to other languageId servers if known.

The existing LSP err is

interface ResponseError<D> {
    /**
     * A number indicating the error type that occurred.
     */
    code: number;

    /**
     * A string providing a short description of the error.
     */
    message: string;

    /**
     * A Primitive or Structured value that contains additional
     * information about the error. Can be omitted.
     */
    data?: D;
}

So would be something like ResponseError<LanguageRange[]>

interface LanguageRange {
    range:Range,
    languageId?:string
}

[aeschli]

@bmewburn please create a separate issue. Please mention which APIs would return that error

[IMarty]

up ?

[nesl247]

Not having this actually renders VS Code useless as a daily editor. I need to be able to work on SQL inside of PHP on a constant basis. It seems like VSCode overlooked a crucial part of any editor early on and is now suffering for it.

[rchrdnsh]

Any news on this? Is there another more preferred and/or updated thread for this issue?

[MattMcFarland]

I wonder if this is difficult to do because we use TextMate? Is this feature in any other TextMate style editor that you know of? I have only seen this with IntelliJ products and I know they don't use TextMate to parse syntax. I wonder if there is something special about that, or if this would require some re-architecting, or perhaps the vs-code team doesn't have bandwidth for it?

[noinkling]

Is this feature in any other TextMate style editor that you know of?

@MattMcFarland Yes, I've seen it in some Sublime Text grammars. For instance if I look at an old Ruby on Rails project of mine with SQL embedded in a heredoc (a.k.a. multi-line string), the string uses SQL highlighting:

dir.up do
  execute <<-SQL
    ALTER TABLE components
      ADD PRIMARY KEY (id),
      ADD CHECK (id ~ '^[A-Z]\\d{1,4}$')
  SQL
end

(edit: looks like even the GitHub highlighting may have some rudimentary support)

The scope tokens (or whatever they're called) for the SQL part look something like this:

source.ruby meta.embedded.block.sql string.unquoted.heredoc.ruby source.sql

In this specific case it's dependent on the heredoc delimiter being SQL, which is a bit limiting. If VS Code adds this feature I'd hope it to be a bit more flexible/configurable (personally I'd accept magic comments if it comes to that).

Edit 2: actually the above example already works in VS Code basically the same as it does in Sublime (I should have checked earlier):

The reason the highlighting doesn't look great is because the SQL grammar file itself isn't great, and because it falls back to the Ruby string colour instead of the normal neutral (i.e. white) foreground colour.

Here's another one with embedded JavaScript instead:

So maybe we could get by with just some grammar updates for other cases (for now)? It might not help with things like intellisense, but at least there will be highlighting. I guess the point in the OP about being "impossible to all hardcode" still stands though.

[MattMcFarland]

TLDR: We already have a mechanism that invokes a language parser, let's modify it so that it can invoke or terminate another language parser when triggered by a token.

How i think language injection works with IntelliJ at least, (prepare for some handwavng) is like this:

1. a token is read by the syntax parser which initiates a context switch of sorts to another language parser entirely.
2. the "other language parser" is then used for the rest of the text
3. another token is read by the "other language parser" which tells it to terminate, bringing you back to the first original language parser.

The architecture/design of VS CODE does not appear to support a feature which allows for it to switch to an entirely different language service or parser when certain tokens are read. Unless this is supported I dont think its possible.

We already have some sort of mechanism that sets the language which I think has to do with file extensions. Now we need to re-use this mechanism or add another one or change this one to allow for the language to be set based on either tokens or lines, but tokens would be ideal.

Instead of having to hardcode the escapes to HTML for every language, it would make a lot more sense to design the software to have the ability to completely switch context of a language to another if some logic is true, then switch back if some other logic is true. What's funny is this is already kinda here, its just based on files instead of syntax. If we could set it up to work off of syntax as well, and isolated the functionality to where it could call itself, you could even do this recursively.

Now because langauge services might add some additional overhead, you could make sure they are sleeping or only use them if the cursor is inside the specific language, or even just scrap that and for the MVP you say "any sub languages" (language parsers invoked by a parent language parser) only use textmate.

But I'm just handwaving here, it probably isnt as simple as I think it is ;)

[benjaminjackman]

FWIW I think maybe this extension sort of manages to pull it off, allowing css to be embedded into typescript / javascript for use with the styled components library (using string template literals)

https://github.com/styled-components/vscode-styled-components

[Lucretia]

Also, for meta-languages like Flex/Bison. Have the major language be Flex/Bison and a minor language be another language, not neceesarily C/C++, as there are implementations in other languages too.

[muuvmuuv]

This would also be nice for automated formatting when you have a PHP and HTML formatter installed, they both should work. The HTML should format the HTML and PHP the PHP inside there tags.

[r-laf]

@noinkling Would you share how to make the VSCode display the debug info for highlighted words? Thanks.

[noinkling]

@zh4ui Developer: Inspect TM Scopes in the command palette.

By the way, I did find an Atom JS grammar that allows this sort of thing for template literals in a customizable way, by using an option where you set pattern -> scope token mappings:

https://github.com/gandm/language-babel#javascript-tagged-template-literal-grammar-extensions

Maybe it can serve as inspiration. That kind of thing would be awesome in VS Code.

[adammichaelwood]

Another important use case I didn't see mentioned above: code snippets in documentation written in Markdown or restructured text. I constantly do:

Here is some paragraph in reStructuredText.

.. code:: python
    def sum(x,y)
        return x + y

Back to my prose here.

[nikita-kiryanov]

Another use case: database stored procedures. Stored procedures can be written in non-SQL languages such as Python, but are defined using SQL statements, such as:

CREATE OR REPLACE FUNCTION schema.foo(
    var1 citext,
    var2 int)
  RETURNS boolean AS
$BODY$
    print 'This is Python code'
    return True
$BODY$
  LANGUAGE plpythonu VOLATILE
  COST 100;
ALTER FUNCTION schema.foo(citext, date, date, date)
  OWNER TO postgres;
GRANT EXECUTE ON FUNCTION schema.foo(citext, int) TO public;
GRANT EXECUTE ON FUNCTION schema.foo(citext, int) TO postgres;
GRANT EXECUTE ON FUNCTION schema.foo(citext, int) TO users;

Basically this is the opposite case of "I have SQL inside my code".

[r-laf]

Another potential use case, c++ 11 raw string literal:

const char* sql = R"SQL(
CREATE TABLE user (
  id INTEGER PRIMARY KEY,
  ...
);
)SQL";

[martixy]

There seems to be a lot of support for this in the thread and very little actual progress.

The idea is nigh on 3 years old now. I would love to stay optimistic, but I think it's safe to say, if it isn't in yet, it's not going in.

[muuvmuuv]

This should work for an Edtior, even Atom can do it, why not VS Code...

[rwatts3]

I didn't know atom had this capability would you be willing to point me to a link.

On Thu, Sep 6, 2018, 12:51 PM Marvin Heilemann notifications@github.com wrote:

This should work for an Edtior, even Atom can do it, why not VS Code...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vscode/issues/1751#issuecomment-419219415, or mute the thread https://github.com/notifications/unsubscribe-auth/AFFsSYcmWdkSf5vT76VA3np1MZphDELmks5uYXyogaJpZM4G9sZE .

--

-V/R

Ryan Watts

[muuvmuuv]

@rwatts3 uff, long time I used Atom, but I know it has worked with atom-beautify extension and a mixed php/html file.

[robhaswell]

Vim has had this feature for years.

On Thu, 6 Sep 2018, 9:14 pm Marvin Heilemann, notifications@github.com wrote:

@rwatts3 https://github.com/rwatts3 uff, long time I used Atom, but I know it has worked with atom-beautify extension and a mixed php/html file.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vscode/issues/1751#issuecomment-419226050, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZW_vJdMGRC0tLsEYrTrYV-sKh4M8Z-ks5uYYI_gaJpZM4G9sZE .

[adammichaelwood]

In Atom it works for code samples is most languages inside a markdown file.

Emacs has polymode, and a few other similar plugins, which allow you to mix modes (with varying degrees of success).

Lots of editors have some version of this feature, and I suppose the writer of any specific language plugin for VS Code could implement it for that plugin.

What I think we need is a set of easy-to-use APIs for plugin authors to be able to implement a multi-language (or embedded language) feature into their work.

The simplest solution (from a plugin author's point of view) would be the ability to define one or more pairs of regex opening/closing tags that define the beginning and end of an embedded language. Then you would want to be able to support all the following cases:

• the opening tag would define the syntax to be used inside the embed section (like <?php or ``python</code> or.. code:: java` all contain the name of the embedded language) - so the regex contains a variable checked by the parser

• different regex pairs define specific languages ( ?> in php is the beginning of an HTML section; <style> is the beginning of a CSS section)

• the opening/closing tags are generic, and the parser guesses by looking at the contents

[Levdbas]

This is the main reason I cannot switch from Atom to Vscode. I would really like to see to see this feature implemented so people can start working on html/php mixed files

[RaccoonFive]

The fact that we don't have this yet makes a lot of templating language syntax extensions like twig or erb pretty bad since there's no native html auto completion with them and most of what you write using these is actually html.

[connorshea]

VS Code's Markdown extension includes support for other languages, though I'm not sure if that includes IntelliSense (e.g. https://github.com/Microsoft/vscode-markdown-tm-grammar/pull/38)