How do I properly wire JavaScript tmGrammer

[stefandobre]

Hey there!

I find myself struggling with replacing the terrible JavaScript syntax highlighting that comes bundled with Monaco.

My starting point was this: https://github.com/microsoft/monaco-editor-samples/tree/master/browser-esm-webpack-monaco-plugin

I only included "typescript" and "css" as languages in the build. Including or excluding typescript doesn't make a difference. I wired up onigasm, monaco-textmate, and monaco-editor-textmate as per your example.

The JavaScript textmate grammar I got from here: https://github.com/textmate/javascript.tmbundle/tree/master/Syntaxes

And this was the result:

This ... can't be right... right?

I'd appreciate any pointers as to where I messed up.

Many thanks in advance!

Here's the sourcecode:

package.json

{
    "name": "monaco",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "private": true,
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "build": "webpack --progress"
    },
    "keywords": [],
    "author": "",
    "devDependencies": {
        "css-loader": "^3.4.0",
        "file-loader": "^5.0.2",
        "monaco-editor": "^0.19.0",
        "monaco-editor-webpack-plugin": "^1.8.0",
        "style-loader": "^1.1.1",
        "webpack": "^4.41.4",
        "webpack-cli": "^3.3.10"
    },
    "dependencies": {
        "monaco-editor-textmate": "^2.2.1",
        "monaco-textmate": "^3.0.1",
        "onigasm": "^2.2.4"
    }
}

webpack.config.js

const path = require("path");
const MonacoWebpackPlugin = require("monaco-editor-webpack-plugin");

module.exports = {
    mode: process.env.NODE_ENV,
    entry: "./index.js",
    output: {
        path: path.resolve(__dirname, "dist"),
        filename: "[name].bundle.js",
    },
    module: {
        rules: [{
            test: /\.css$/,
            use: ["style-loader", "css-loader",],
        }, {
            test: /\.ttf$/,
            use: ['file-loader']
        }],
    },
    plugins: [
        new MonacoWebpackPlugin({
            languages: ["typescript", "css"],
        })
    ]
};

index.js

import * as monaco from "monaco-editor/esm/vs/editor/editor.api";
import { loadWASM } from 'onigasm' // peer dependency of 'monaco-textmate'
import { Registry } from 'monaco-textmate' // peer dependency
import { wireTmGrammars } from 'monaco-editor-textmate'

// create div to avoid needing a HtmlWebpackPlugin template
const div = document.createElement('div');
div.id = 'root';
div.style = 'width:800px; height:600px; border:1px solid #ccc;';

document.body.appendChild(div);

(async function(){
    await loadWASM('onigasm/lib/onigasm.wasm') // See https://www.npmjs.com/package/onigasm#light-it-up

    const registry = new Registry({
        getGrammarDefinition: async (scopeName) => {
            return {
                format: 'plist',
                content: await (await fetch('static/grammars/JavaScript.plist')).text()
            }
        }
    })

    // map of monaco "language id's" to TextMate scopeNames
    const grammars = new Map()
    grammars.set('javascript', 'source.js')
    //grammars.set('typescript', 'source.ts')

    monaco.languages.register({id: 'javascript'});
    await wireTmGrammars(monaco, registry, grammars)

    monaco.editor.create(
        document.getElementById('root'),
        {
            value: [
                'var abc = 123;',
                'abc++;',
                'console.log(abc);'
            ].join('\n'),
            language: 'javascript',
            theme: 'vs-dark'
        }
    );

})();

[zikaari]

did you check your console for any errors? This might be the point of failure:

await loadWASM('onigasm/lib/onigasm.wasm')

make sure that you get HTTP 200 for .wasm file because webpack needs to be taught on how to "bundle" wasm files properly. Let me know and then we can look further.

[stefandobre]

Thanks for the response!

There are no console errors. onigasm.wasm is loading properly... For now, I just copied the file and included it in my /dist folder manually.

What I did change in the meantime is take my grammar file from the vscode-textmate repo, as opposed to the textmate repo. https://github.com/microsoft/vscode-textmate/tree/master/test-cases/themes/syntaxes

This seems to make a bit of a difference:

but it's still not great.

I also thought that as JavaScript is based on TypeScript, I should include the TypeScript grammar as well. So I'm not bundlig Monaco with Typescript, but including it like this:

    await loadWASM('onigasm/lib/onigasm.wasm') // See https://www.npmjs.com/package/onigasm#light-it-up

    const registry = new Registry({
        getGrammarDefinition: async (scopeName) => {
            if(scopeName == 'source.ts'){
                return {
                    format: 'json',
                    content: await (await fetch('static/grammars/TypeScript.tmLanguage.json')).text()
                }
            } else if (scopeName == 'source.js'){
                return {
                    format: 'json',
                    content: await (await fetch('static/grammars/JavaScript.tmLanguage.json')).text()
                }
            } else {
                return null;
            }

        }
    })

    // map of monaco "language id's" to TextMate scopeNames
    const grammars = new Map();
    grammars.set('typescript', 'source.ts');
    grammars.set('javascript', 'source.js');

    monaco.languages.register({id: 'typescript'});
    monaco.languages.register({id: 'javascript'});
    await wireTmGrammars(monaco, registry, grammars)

Aaaand nothing changed. Same result.

[zikaari]

can you share your repo so I can add some breakpoints and see what's going on?

[stefandobre]

Here it is: https://github.com/stefandobre/browser-esm-webpack-monaco-plugin

[zikaari]

Everything is rigged perfectly except the fact the because you're using TM grammars, the code is getting tokenized with much more "resolution" than what default themes of monaco can handle, i.e monaco editor isn't "prepared" to deal with complex tokens that tm grammars produce.

To fix, that you MUST also provide monaco with an "advanced" theme that has colors for those complex tokens.

Also you should pass the editor instance to wireTmGrammars.

Read more about that here - https://github.com/Nishkalkashyap/monaco-vscode-textmate-theme-converter#monaco-vscode-textmate-theme-converter

You can either convert themes on the fly, or in advance (JIT or AOT, as you see fit)

Working example -

    monaco.editor.defineTheme('vs-code-theme-converted', {
        // ... use `monaco-vscode-textmate-theme-converter` and pass the parsed object here
    });

    window.editor = monaco.editor.create(
        document.getElementById('root'),
        {
            value: [
                'var abc = 123;',
                'abc++;',
                'console.log(abc);'
            ].join('\n'),
            language: 'javascript',
            theme: 'vs-code-theme-converted'
        }
    );

    await wireTmGrammars(monaco, registry, grammars, window.editor)

Example VS code theme - https://github.com/thomaspink/vscode-github-theme/blob/master/themes/github.json

[zikaari]

I've updated the docs (README) regarding the above caveat, I had to re-run the build before releasing 2.2.2, can you please re-install monaco-editor-textmate@2.2.2 and see if everything's nominal?

see https://github.com/NeekSandhu/monaco-editor-textmate/commit/e8c5c506f9e158f10cfacb1a14916fc26a3704d5

[stefandobre]

Thanks so much for your help! Everything works now as expected, also with 2.2.2

[ChristopherHButler]

https://github.com/stefandobre/browser-esm-webpack-monaco-plugin

Hey @stefandobre Could you share this repo? I am struggling to do the same thing and I would greatly appreciate any help!

[stefandobre]

Not sure what happened to that repo, but my implementation does not use this technique anymore, even though it did work initially! In the end I resorted to using a simplified version of this: https://github.com/bolinfest/monaco-tm You can find the relevant bits here: https://github.com/foex-open-source/apex-builder-extension-by-fos/blob/master/src/editor/lib/monacoEditorHelper.js (The code is a mess and not really meant for public consumption 😅)

[ChristopherHButler]

Something is better than nothing! I'm struggling to get it working at all. Do you know if the package you referenced can work with react? Maybe I can look into the source 🧐

[stefandobre]

Not sure. I wouldn't recommend using the package though. From what I remember it's pretty restricted. You could try to implement the same logic yourself though. No reason why it shouldn't work in react. But yeah... this whole monaco & tm grammar business gave me quite some headaches too. Good luck!

[ChristopherHButler]

Ok I think I am close or at the point you were when @NeekSandhu wrote: To fix, that you MUST also provide monaco with an "advanced" theme that has colors for those complex tokens.

I was trying to follow the instructions from https://github.com/Nishkalkashyap/monaco-vscode-textmate-theme-converter#monaco-vscode-textmate-theme-converter But I am not sure I have the correct theme now.

Would it be too much trouble for you to take a peek at my repo? vscode-themes-in-monaco

The only file of interest is here: https://github.com/ChristopherHButler/vscode-themes-in-monaco/blob/master/src/components/CodeEditor.js 😁 🙏🏼

[ChristopherHButler]

Hey @stefandobre never mind, I got it working!! 🥳

[golmujik]

When tokenization works service workers dont please help somebody

[golmujik]

Hii this is a silly question but I wanna ask.

How to verify if tm is working?

Can i do so by checking if the theme and colored words in editor is showing without including any language in webpack? does this means it is working? please help

[golmujik]

And when i remove this line

await wireTmGrammars(monaco, registry, grammars, editor);

the coloring stops. So this means tm is working properly right?

[golmujik]

coloring stops means only few words get highlighted after that

or it should not even color few words if i remove this line

await wireTmGrammars(monaco, registry, grammars, editor);

[zikaari]

Can i do so by checking if the theme and colored words in editor is showing without including any language in webpack? does this means it is working? please help

If you excluded the language from webpack, and ALSO from monaco-editor-textmate linking, no colors should appear at all.

Colors should come back to life once you re-link the grammar through monaco-editor-textmate.

[golmujik]

If you excluded the language from webpack, and ALSO from monaco-editor-textmate linking, no colors should appear at all.

Yes when I use import * as monaco from "monaco-editor/esm/vs/editor/editor.api"; with webpack with languages: [] All color goes away.

Colors should come back to life once you re-link the grammar through monaco-editor-textmate.

And comes back as per the theme when I use the wireTmGrammars

But when I use import * as monaco from "monaco-editor/esm/vs/editor/editor.api"; the tm tokens seems to work but all other features vanish like intellisense autocomplete, etc. and I check the console it shows no service workers being registered even after adding this

window.MonacoEnvironment = {
  globalAPI: true,
  getWorkerUrl(moduleId, label) {
    console.log(label);

    if (label === "json") {
      return "./dist/json.worker.bundle.js";
    }
    if (label === "css") {
      return "./dist/css.worker.bundle.js";
    }
    if (label === "html") {
      return "./dist/html.worker.bundle.js";
    }
    if (label === "typescript" || label === "javascript") {
      return "./dist/ts.worker.bundle.js";
    }
    return "./dist/editor.worker.bundle.js";
  },
};

And when I use this line import * as monaco from "monaco-editor"; All features comes back to normal but the tm tokens then sometimes work and sometimes don't

Conclusion is when I use import * as monaco from "monaco-editor/esm/vs/editor/editor.api"; as per the guide of this repo the tokenization seems to work very well but as mentioned all other features vanish like intellisense autocomplete, etc.

So is there a way to make those features work back properly. I mean the workers dont even get registered If I use import * as monaco from "monaco-editor/esm/vs/editor/editor.api";

Please help

[golmujik]

Webpack

const MonacoWebpackPlugin = require("monaco-editor-webpack-plugin");

const path = require("path");

module.exports = {
  entry: {
    app: "./index.js",

    "editor.worker": "monaco-editor/esm/vs/editor/editor.worker.js",
    "json.worker": "monaco-editor/esm/vs/language/json/json.worker",
    "css.worker": "monaco-editor/esm/vs/language/css/css.worker",
    "html.worker": "monaco-editor/esm/vs/language/html/html.worker",
    "ts.worker": "monaco-editor/esm/vs/language/typescript/ts.worker",
  },
  output: {
    globalObject: "self",
    filename: "[name].bundle.js",
    path: path.resolve(__dirname, "dist"),
  },
  module: {
    rules: [
      {
        test: /\.css$/,
        use: ["style-loader", "css-loader"],
      },
      {
        test: /\.ttf$/,
        use: ["file-loader"],
      },
    ],
  },
  plugins: [
    new MonacoWebpackPlugin({
      languages: [],
    }),
  ],
  mode: "development",
  resolve: {
    fallback: {
      path: require.resolve("path-browserify"),
    },
  },
};

[golmujik]

index.js

// import * as monaco from "monaco-editor";
// or
import * as monaco from "monaco-editor/esm/vs/editor/editor.api";

// if shipping only a subset of the features & languages is desired

import { loadWASM } from "onigasm"; // peer dependency of 'monaco-textmate'
import { Registry } from "monaco-textmate"; // peer dependency
import { wireTmGrammars } from "monaco-editor-textmate";

window.MonacoEnvironment = {
  globalAPI: true,
  getWorkerUrl(moduleId, label) {
    console.log(label);

    if (label === "json") {
      return "./dist/json.worker.bundle.js";
    }
    if (label === "css") {
      return "./dist/css.worker.bundle.js";
    }
    if (label === "html") {
      return "./dist/html.worker.bundle.js";
    }
    if (label === "typescript" || label === "javascript") {
      return "./dist/ts.worker.bundle.js";
    }
    return "./dist/editor.worker.bundle.js";
  },
};

async function liftOff() {
  await loadWASM(`https://cdn.jsdelivr.net/npm/onigasm@2.2.5/lib/onigasm.wasm`); // See https://www.npmjs.com/package/onigasm#light-it-up

  const registry = new Registry({
    async getGrammarDefinition(scopeName) {
      if (scopeName === "source.js") {
        return {
          format: "json",
          content: await (await fetch("./JavaScript.tmLanguage.json")).text(),
        };
      }

      if (scopeName === "source.ts") {
        return {
          format: "json",
          content: await (await fetch("./TypeScript.tmLanguage.json")).text(),
        };
      }

      return {
        format: "json",
        content: await (await fetch("./TypeScript.tmLanguage.json")).text(),
      };
    },
  });

  // map of monaco "language id's" to TextMate scopeNames
  const grammars = new Map();

  monaco.languages.register({ id: "javascript" });
  monaco.languages.register({ id: "typescript" });

  grammars.set("javascript", "source.js");
  grammars.set("typescript", "source.ts");

  monaco.editor.defineTheme("vs-code-theme-converted", {
     //Theme data
  });

  // await wireTmGrammars(monaco, registry, grammars);

  const editor = monaco.editor.create(document.getElementById("container"), {
    value: [`console.log('Hello')`].join("\n"),
    language: "javascript", // this won't work out of the box, see below for more info,
    theme: "vs-code-theme-converted", // very important, see comment above
  });

  await wireTmGrammars(monaco, registry, grammars, editor);

  // monaco.editor.setTheme("custom-theme");
}

liftOff();

This seems to work perfectly but other features stops and only this editorServiceWorker gets registered but other dont get registered nor downloaded.

[golmujik]

@NeekSandhu Please help me I have been searching solutions since 3 days 😢

[relliv]

A few years later, thanks to everyone who commented. I leave a small stackblitz demo link. It may be useful for curious people 👍

[ChristopherHButler]

I think I did make a working demo of this as well. I will share the link here in case it can still help someone out: https://github.com/ChristopherHButler/vscode-themes-in-monaco