webpack/browserify compatible AMD headers

[NikGovorov]

Hello,

Is there any way to change AMD headers to be compatible with webpack/browserify?

Are there any plans to support CommonJS, UMD or ECMAscript 6 for the package and external modules in monaco.d.ts?

[nescalante]

@TheLarkInn, looking forward for your implementation for webpack. Let me know if I can do something to help on accelerating the development of that since now I am blocked due this issue.

[ThorstenBux]

I just came across this one here: https://www.npmjs.com/package/monaco-loader Haven't tested it myself but maybe someone finds it helpful.

[mjkkirschner]

we use require.js to bundle our application, I have tried various ways of including the minified editor-main.js file using require - and it does correctly get bundled, even executed, but monaco is left undefined - is it true that you can use require.js somehow to load monaco? apologies if I should start a different issue.

[ev45ive]

Benefit of using standard module loading convention in your code is that it allow for moving all loading/bundling logic to be placed "outside of your code". That is - in bundler / loader config.

All popular loaders handle (via plugins) things like importing css, translation, lazy loading of code and bundling / code splitting.

Monaco is great piece of software and there is lot potential for it, especially when used as part of specialised software.

IMHO the goal here is not only to make it easy to just drop in but also to make these things configurable for integrator who needs custom stuff. Doesnt have to be weback, but Webpack is perfect for that, making your code agnostic of way it its built and consumed. Everything is just a module (TM).

Some modules are sync, some are asyc (webpack handles that also), some are css or strings or internationalization strings(!).

Best part is that your code doesnt care. Loader does. You just import strings. Thats why its important for people i guess.

Any progress on this.btw?' :-)

[kube]

@rebornix @TheLarkInn Sorry for the ping, but is this something that will come in a near future? It seems the idea was abandoned.

[ev45ive]

Where can one find details on loader and i18n stuff. Maybe we could work on getting plugins for webpack so that integrator can decide which modules are bundled and which are lazy loaded, but all with their (integrators) own bundling system.

I Guess the main issue here is that i have 2 loaders now: 1- webpack - witch i have full control over (configuration) 2- monaco custom require which i cannot configure

also lazy loading, assets and internationalization loading logic is duplicated then.

[ev45ive]

@TheLarkInn - is there maybe an option to "capture" calls to require in loaded modules and put in there custom logic? Especially for async requires or dynamic ones like monaco uses. Sorry for bother, but Can't find it in webpack docs.

[timkendrick]

@alexandrudima @rebornix @TheLarkInn I decided to have a crack at this over the weekend seeing as there doesn't seem to have been much progress on this issue lately.

TL;DR: I've made a proof of concept that uses Webpack to successfully compile the editor/plugins/CSS/etc into a single massive UMD script with no dependencies (published as @timkendrick/monaco-editor). It inlines CSS and remote worker scripts into the client-side bundle, and if you're feeling adventurous you can create a custom build using the Webpack loader I wrote. In short: it's totally possible to bundle Monaco via Webpack without any upstream changes.

In-depth technical details:

While this method seems to work fine, it uses a fairly elaborate webpack setup internally and I'm by no means suggesting that any of it is remotely sane. I'll try and explain the approaches I explored and suggest potential alternative avenues as I go along.

It was a bit of an ordeal trying to get this working: most of the time went into writing a custom Webpack loader (@timkendrick/monaco-editor-loader), which wraps the AMD modules with an injected RequireJS shim that gets executed at runtime and stitches all the modules together, taking into account the various extensions you've loaded etc. Ideally you'd just be able to run it through webpack and have done with it, or failing that I would have preferred to at least use something off-the-shelf like Almond, but the CSS and web worker parts of the Monaco code rely on require.toUrl() which isn't supported by Webpack or Almond, so I ended up writing my own custom shim. This started off as a few lines, but kept growing as more features were needed - looking back now it might have been better to investigate adding toUrl() behavior to Almond or something. It's several years since I've used RequireJS and the API docs are pretty vague, so I'm not super-confident that my loader is 100% compatible with the Monaco loader.js, but I basically just kept adding features till everything worked… it seems to get it all done without any problems though.

A nice benefit of going down the webpack route is that it gives you a lot more control than RequireJS over how you want to bundle your assets. As well as supporting async asset loading via baseUrl, my loader allows the user to choose to inline specific assets (e.g. editor.main.css, workerMain.js etc) into the main bundle, so that you don't need to serve these separately. Any CSS assets you specify will be inlined as data: URIs and any worker scripts will be inlined as blob: URIs (and loaded via URL.createObjectURL(blob)). This went surprisingly smoothly. Somewhat trickier was the fact that workerMain.js contains its own inbuilt version of the AMD loader, which asynchronously loads in any remote plugin worker scripts from within the worker process. Obviously we don't want this behaviour if we've already downloaded the remote scripts in the main bundle, so instead you need to inject them into the worker script through some other means. I ended up writing another webpack loader that mocks the importScripts() function exposed to the bundled worker script (in this case, workerMain.js), inlining the additional bundled remote scripts as part of the main worker script blob (which itself is inlined into the main bundle), thereby allowing you to combine all the additional remote worker scripts into the same local bundle, along with the main JS and CSS. If you choose not to bundle some or all of the assets, it'll fall back to the default behavior of loading them from the baseUrl (both in the main process and in the worker process).

Bearing all this in mind, If I were to do this again I imagine a much cleaner overall approach would be to run Webpack on the source TypeScript files directly. This isn't possible as it stands, seeing as Monaco relies on the additional AMD require() properties that aren't present on the node/webpack require() function, so Webpack will complain about being used dynamically (as far as I can gather, in Webpack require() operates more like a keyword than an object), so anything that relies on the extra AMD stuff will cause the build to break. Perhaps the least dirty approach would be to apply some kind of an AST transformation to all the Monaco source files at compile time, replacing require.toUrl() calls with an inlined data: or blob: URL containing the target asset contents instead? I feel like it's either that, or monkey-patch Webpack's require() somehow while still maintaining the usual Webpack require behaviour (not sure if that's even possible without an AST transformation). There's also the issue of the vs/css!... and vs/nls!... plugin paths, but I suspect these would be pretty simple to solve via style-loader, and maybe some kind of custom i18n loader, I haven't really looked into it. As far as I'm aware though, assuming you set up a vs path resolver alias and equivalent loaders for the vs/css and vs/nls plugins, the only real impediment to using Monaco directly within Webpack is the use of require.toUrl/require.config etc., so if you can get around this it should be plain sailing.

Anyway, this was just an experiment, so I wouldn't recommend using my package in production: I haven't looked into browser compatibility / proper sourcemapping etc., and I'm not planning to maintain it, hence the @timkendrick/ prefixes) – but at least it proves it's possible. Hopefully this will be a useful starting point if others want to pick it up from here.

Give me a shout if you want more details on anything, I'm happy to help!

[TheLarkInn]

@timkendrick Thanks for this awesome write-up.

I would love to see the possibility of having all of Monaco/code bundled with webpack. For one, webpack's ability to bundle electron apps typically result in over 40% start time increase (so theres a legit win there). This would also let one convert most of the existing modules to be ESM.

One of my concerns is that there is a set of dynamic nature that Monaco relies on for loading plugins and services. Because webpack is a static build tool, there would have to be some separation between the core vscode functionality, and the extension system and providers, etc.

externals[vscode-extension-runtime: ?] => dynamicExtentionProvider
[vscode core code: esm]

This is kind of off the cuff and not well thought out at the moment, but I think there's a lot of ROI (build perf, general Monaco adoption, code launch and runtime perf), that could be gained from the transition.

[timkendrick]

@TheLarkInn yep totally agree – my runtime AMD-style shim was just a hack to get it working, I'd much prefer if it were 100% webpack.

As for the plugin-loading system, I've been facing the same problem in a pluggable Electron webpack project I'm working on – I've ended up with the core bundle exposing a global registerPlugin() style method on the window object, which each of the extension bundles calls at runtime when they start up (this feels a bit like it's not making the best use of the bundler, but it works fine I guess).

I might be misunderstanding what you're saying, but, isn't Monaco already essentially performing this same process via the global monaco api (see snippet)? If that's the case then I can't imagine how webpack would pose any problems here…

https://github.com/Microsoft/monaco-typescript/blob/fd172013f77fa599e03a1c17b79aab62637569fa/src/monaco.contribution.ts#L194-L222

[felixfbecker]

Not sure if this helps, but it's also possible to use dynamic import() with webpack, which works well with TypeScript too.

In general, I think the best way is to have Monaco just use ES6 imports everywhere and the consumers can decide whether they want to bundle everything, let webpack figure out good chunks to load async or to transpile everything to AMD.

[timkendrick]

@TheLarkInn I had a chance to take another look at this over the Christmas break, and I've managed to get Monaco compiling entirely via webpack from the original TypeScript source files, with full source map support. It's a lot cleaner than my previous attempt, but there are still a few hacks and workarounds to get everything playing nicely together.

TL;DR: head to the @timkendrick/monaco-editor npm package if you want to grab a batteries-included equivalent of the official monaco-editor package (includes a standalone build containing everything inlined into a single JS file, as well as a build that consists of separate JS, CSS and worker JS files).

How it works

• Webpack seems to have trouble loading pre-built VS Code UMD bundles, so I've had to pull in all the additional libraries as Git submodules and rebuild all the extensions from source as well

• The webpack config defines aliases for all the bundled VS Code modules (vs, vs/basic-languages etc) so that imports are wired up correctly

• Some of the modules use a runtime require() call to load additional modules; the webpack config defines individual contexts for each of these modules to effectively inline a predetermined set of additional modules

• The main entry point (editor.main.ts) is run through my custom @timkendrick/monaco-editor-loader webpack loader, which bundles up the main entry point script along with the extension entry point scripts, plus the entry points for the additional worker scripts

• All the TypeScript source files are run through ts-loader with the VS Code tsconfig.json settings plus a couple of additional overrides - notably module: "commonjs", rather than module: "amd" ("es6" seems to present problems…)

• Once they have been transpiled to JS, the source files are run through a couple of custom Babel transforms (defined in @timkendrick/babel-preset-vscode), which a) rewrite the AMD-style require() calls to CommonJS style, and b) convert any vs/css!foo imports to foo.css

• Any CSS imports are siphoned off by the style-loader / ExtractTextWebpackPlugin and handled via webpack

• Worker scripts are either bundled into the main output file as blob: URLs or emitted as separate files and loaded by the browser (custom webpack loaders are used to implement both of these mechanisms)

• I haven't looked into supporting the vs/nls stuff correctly; the default language seems to work fine out-of-the-box but I suspect that adding proper i18n support would entail writing another fairly simple custom loader

• The webpack setup currently requires webpack 4.x due to a bugfix that I submitted for an issue that was breaking the webpack 3 build

All in all, this is still a pretty complex webpack setup but it feels a lot more robust than my previous attempt, and as predicted, startup performance is much better. To me, this approach feels stable enough to use in an actual project.

I'd say based on this experiment that if the VS Code team felt that webpack compatibility was a desirable goal for the Monaco editor then it probably wouldn't take long to convert the codebase - there are a few dynamic require() call expressions that could be rewritten statically to avoid the need for webpack context plugins, and you'd need to use webpack loaders as a replacement for the AMD-dependent require.toUrl() runtime worker loading and for the vs/nls stuff, but apart from that it should all be pretty trivial. I haven't had much experience with webpack's code splitting, but I imagine this could be very handy if there was a need for a minimal core and async extension loading.

This was all done as a proof-of-concept so I can't guarantee that I'll be able to maintain these npm packages. If anyone feels like investigating what would need to be done to adapt the main VS Code codebase for webpack, I'd recommend looking at timkendrick/monaco-editor/webpack/createConfig.js to see how I worked around the various edge cases, and timkendrick/monaco-editor-loader/lib/loaders/monaco.js to see how it compiles the main entry point and bundles the workers.

Let me know if there's any more info I can provide on this!

[mnpenner]

Thanks @timkendrick! Works great.

I wrapped it in a React component. Here's the gist if anyone's looking for a quick copy-paste solution:

import React from 'react';
import * as monaco from '@timkendrick/monaco-editor';

export default class MonacoEditor extends React.Component {

    componentDidMount() {
        monaco.editor.create(this.el, {
            language: 'javascript',
        });
    }

    render() {
        return <div style={{width:'800px',height:'600px',border:'1px solid grey'}} ref={n=>this.el = n}/>
    }
}

[alexdima]

Thank you for your patience! :heart:

---

I am happy to let you know that we have just published v0.11.0, which is the first release to include the editor packaged in an ESM format under the esm folder. This pretty much means that the editor can be consumed as simple as:

import * as monaco from 'monaco-editor';

There are 2 complete samples of using the editor with webpack that I have put together:

---

1. "Basic"

Full working example here.

The integration burden consists of:

• You must package web workers. See webpack.config.js
• You must let the runtime know where you have packaged those web workers. See index.js
  • You must ignore require() calls in typescriptServices.js when packaging. See webpack.config.js

2. "Picky"

Full working example here.

Same integration burden as above, but this example loads only the core of the editor, the find widget, the python colorization and nothing else! See index.js. This would allow you to pick which editor features you want in order to get a smaller bundle size.

[TheLarkInn]

OMG! WOW! This is awesome! I'll be sure to try this out and provide feedback.

[rockymontana]

Hello and nice work! I've been trying to get this to work with webpack today. I've come as far so that I can see the editor, but I get a lot of console errors:

(I especially like the "window not found" error ;)

Anyways. I mainly wonder three thing:

1. What does self point to in self.MonacoEnvironment? I don't see any references to self and when I've been poking around in the sources it seems like it should be pointing to window- is that correct?

2. The workers in MonacoEnvironment object points to ./some.worker.js. Would it be just as fine to link to something like window.location.href + 'editor.worker.js'?

3. Is it important that the webpack config keeps the workers in seperate files rather than bundling all files to the same script?

Sorry if my questions are ridicolous, I'm still kinda new to this. :-)

[timkendrick]

@rockymontana did you add the webpack ignore plugin as specified in the final step of the "Basic" instructions above?

[rockymontana]

Yes I did.

entry: {
    app: './src/main.js',
    "editor.worker": './node_modules/monaco-editor/esm/vs/editor/editor.worker.js',
    "json.worker": './node_modules/monaco-editor/esm/vs/language/json/json.worker',
    "css.worker": './node_modules/monaco-editor/esm/vs/language/css/css.worker',
    "html.worker": './node_modules/monaco-editor/esm/vs/language/html/html.worker',
    "ts.worker": './node_modules/monaco-editor/esm/vs/language/typescript/ts.worker',
  },
  output: {
    path: config.build.assetsRoot,
    filename: '[name].js',
    publicPath: process.env.NODE_ENV === 'production'
      ? config.build.assetsPublicPath
      : config.dev.assetsPublicPath
  },
  plugins: [
    // Ignore require() calls in vs/language/typescript/lib/typescriptServices.js
    new webpack.IgnorePlugin(
      /^((fs)|(path)|(os)|(crypto)|(source-map-support))$/,
      /vs\/language\/typescript\/lib/
    )
  ],

Like that

[TheLarkInn]

@rockymontana I'd add html-webpack-plugin and remove the default index.html with one that can be generated automatically to inject bundles for you.

[rockymontana]

The bundles are automatically injected. I see them in the markup. The rest of what you said was above my level of understanding 😭

[corbinu]

@alexandrudima I believe #40 #60 can now be closed also

[alexdima]

@rockymontana

The challenge with the editor is that is creates web workers. Web workers have a completely different JS context, i.e. they are a fresh runtime. Also, web workers must behave differently than the source code in the UI thread. In the UI thread, our sources set up a monaco API which you can then use in order to create an editor. In the worker thread, our sources begin listening to messages and reacting to them correctly, according to a protocol which we define between the UI thread and the workers.

@timkendrick has managed to bundle the worker code as a blob in the bundle (probably similar to how CSS can be included as a blob), but my limited webpack experience prevented me from setting up something so cool :).

The error you are running into is IMHO a configuration error. I believe you are not defining the getWorkerUrl correctly in your application. i.e.:

self.MonacoEnvironment = {
    getWorkerUrl: function (moduleId, label) {
        if (label === 'json') {
            return './json.worker.bundle.js';
        }
        if (label === 'css') {
            return './css.worker.bundle.js';
        }
        if (label === 'html') {
            return './html.worker.bundle.js';
        }
        if (label === 'typescript' || label === 'javascript') {
            return './ts.worker.bundle.js';
        }
        return './editor.worker.bundle.js';
    }
}

The UI thread talks 100% the same way to the web workers, both in the AMD and the ESM case. In the AMD case, a web worker is loaded and then it gets specialized into a language worker via loadForeignModule. In the ESM case, we load a web worker which is directly specialized and then it will ignore the loadForeignModule message, as it has already that module loaded. So I believe there is a configuration problem in your code where the non-specialized worker is loaded for a language worker.

TL;DR put a breakpoint in your getWorkerUrl implementation and make sure you return the correct URL. Also, if you are creating your own web workers, be sure to also cover those in that method.

---

@TheLarkInn @timkendrick I am a webpack noob and this might be obvious :). When you get a chance, can you please take a look at the two examples and let me know if there are easier ways to set things up for webpack or if there are more things we can do in the way we distribute the ESM source code to make things easier.

[OneCyrus]

wow thanks! that's great.

@TheLarkInn wouldn't it be possible to use dynamic imports to generate seperate bundles (files) for the workers?

[rockymontana]

@alexandrudima Thanks a lot for the answer! I myself is quite the webpack noob so it's probably me doing something silly. The first thing I see is that you set self.MonacoEnvironment as where I set it to window.monacoEnvironment. The reason why I did that is so that I'm able to use the bundle where I need it instead of loading it globally even though I might not need it.

The second thing is that I try to use your config with my project which is a VueJS project. I think I understand what has to be done, so I'll give it another shot and if I get lucky I'll see if I can explain what and why. Again - Thanks for the help!

[corbinu]

@rockymontana I am trying to get it up in a vue app also if you want to collaborate.

[mnpenner]

Would require.context be of use here? It would force webpack to bundle all the workers without having to create separate entry points for each one, and should let you pull the paths out so you can load them into workers so we don't have to implement getWorkerUrl either, if I'm not mistaken.

[timkendrick]

@mnpenner it turns out it's not quite as simple as that - see discussion on #759

[rockymontana]

@corbinu Absolutely! How would you like to do that?

[corbinu]

@rockymontana Well at the moment my version is in a private Gitlab repo. I can give you access or try to open source it later today. If it is easier your welcome to drop me an email

[rockymontana]

What’s your email?

[moranje]

@rockymontana If you do find a solution for vue, do share. I'm trying to set this up in a CodeSandBox vue environment but I'm having trouble figuring this out as well. See the editor component if you are curious.

[rebornix]

@moranje it might be because that Codesandbox does the bundling for you, which you can't choose the entry file (then no way to set up the worker). I'll sync with Ives and see if we can fix this problem or set up a Monaco editor template.

[corbinu]

edit: removed email

I will note mine is using the new vue-cli 3 beta

[rockymontana]

I really don't understand how I should configure the getWorkerUrl.

If you look at the following screenshots you'll see that I can retrieve the workers, and I can use the MonacoEnvironment from the console:

As you see in the screenshot above I can do this: MonacoEnvironment.getWorkerUrl('', 'html') which returns: "http://127.0.0.1:8080/html.worker.js"

And the "proof" that it's reachable:

If I can configure this wrong, could you explain it to me like I'm a five year old, because I really don't see what's being misconfigured here.

Here's the webpack config again:

  entry: {
    vue: './src/main.js',
    app: './src/monaco.js',
    "editor.worker": './node_modules/monaco-editor/esm/vs/editor/editor.worker.js',
    "json.worker": './node_modules/monaco-editor/esm/vs/language/json/json.worker',
    "css.worker": './node_modules/monaco-editor/esm/vs/language/css/css.worker',
    "html.worker": './node_modules/monaco-editor/esm/vs/language/html/html.worker',
    "ts.worker": './node_modules/monaco-editor/esm/vs/language/typescript/ts.worker',
  },
  plugins: [
    // Ignore require() calls in vs/language/typescript/lib/typescriptServices.js
    new webpack.IgnorePlugin(
      /^((fs)|(path)|(os)|(crypto)|(source-map-support))$/,
      /vs\/language\/typescript\/lib/
    )
  ],

And the generated markup from my index-file (at least the body parts):

    <script type="text/javascript" src="/vue.js"></script>
    <script type="text/javascript" src="/css.worker.js"></script>
    <script type="text/javascript" src="/html.worker.js"></script>
    <script type="text/javascript" src="/json.worker.js"></script>
    <script type="text/javascript" src="/ts.worker.js"></script>
    <script type="text/javascript" src="/editor.worker.js"></script>
    <script type="text/javascript" src="/app.js"></script>

Does that give you any ideas on why it's failing?

Update - Added Plugin section from the webpack config

[meyer]

@rockymontana i don’t think the web worker files should be included in a script tag. The editor will create a new worker using the URLs you provide. You probably need to customise the template to exclude web worker files.

this probably belongs in a new issue btw. the self.require issue is being discussed in #759 and the remaining issues appear to be related to web workers being loaded in a non-web worker environment.

[rockymontana]

@meyer I tried running the webpack example to see whether or not that would generate the workers as script tags, and just as you say, they actually don't. However - I can't really see anything that differs in the config, but then again - I suck pretty hard at webpack, so who am I to say hehe.

However - I checked the debugger dev tool tab and I can see that ts.worker.js is defined as a worker there, so I guess the browser can figure out that it's a worker even though it's a script tag. ¯_(ツ)_/¯

[moranje]

@rebornix @CompuIves thanks, that'll be great

[corbinu]

I opensourced what I have so far here. It is a bit of a mess but everybody feel free to make experimental PRs or I am happy to add people as developers. I will move it to github as soon as I have time to clean it up and write a readme.

https://gitlab.com/corbinu/monaco-editor/

[TheLarkInn]

@corbinu Can I recommend that for specific changes you break them up and submit PR's to the source? This can help ensure that changes are tracked and understood and the Diff's can be reviewed etc.

[TheLarkInn]

And have issues that tie to said PR's it would be nice to have people discussion implementation details and whatnot in a shared context like a code review (even if it doesn't get merged, it still helps identify all the things that have to be cleaned up for this to work really well!)

[alexdima]

@timkendrick has created a webpack plugin that hides all of this config away in PR #767 , and adds cool options to be able to pick whichever languages you want to use, or whichever features you want to use.

I will create a new editor release with this plugin shortly, and I hope that will simplify things tremendously.

[ngohungphuc]

Hi guys. Sorry to bother but I have write a blog post here to show how to integrate Monaco editor into Angular 7.