Using monaco workers for language support

[rubenfiszel]

Hi,

I'm trying to upgrade from a previous version of the monaco-languageclient.

I'm only interested in the language client and otherwise prefer to load monaco than vscode, especially for languages that we are not supported by vscode at the moment.

To load workers for client-side languages, I previously used:

import type { Environment } from 'monaco-editor/esm/vs/editor/editor.api.js'
import cssWorker from 'monaco-editor/esm/vs/language/css/css.worker?worker'
import htmlWorker from 'monaco-editor/esm/vs/language/html/html.worker?worker'
import tsWorker from 'monaco-editor/esm/vs/language/typescript/ts.worker?worker'
import jsonWorker from 'monaco-editor/esm/vs/language/json/json.worker?worker'
import editorWorker from 'monaco-editor/esm/vs/editor/editor.worker?worker'

and an extension such as: https://www.npmjs.com/package/monaco-graphql which assume some monaco contribs are there.

With the new version of monaco-languageclient this seems to not be possible anymore because my understanding is that it doesn't use monaco but vscode instead.

How can I support those languages as I did previously ? I see there are vscode language default extensions that could support most of the languages except graphql, but it seems to me that it requires textmate and that it's heavier than the usual monaco language support.

Our use-case is for windmill (https://github.com/windmill-labs/windmill) and users close and start editors a lot, as soon as they change nodes in the app or flow editor. Hence, I would prefer a more lightweight situation.

I'm currently stuck unable to upgrade.

[CGNonofr]

Hi,

I'm only interested in the language client and otherwise prefer to load monaco than vscode

You can't really separate monaco from VScode, that's the same things, VSCode is built around monaco

especially for languages that we are not supported by vscode at the moment

I would be surprised there is a language supported by monaco but not VSCode, do you have an example?

To load workers for client-side languages, I previously used:

import type { Environment } from 'monaco-editor/esm/vs/editor/editor.api.js'
import cssWorker from 'monaco-editor/esm/vs/language/css/css.worker?worker'
import htmlWorker from 'monaco-editor/esm/vs/language/html/html.worker?worker'
import tsWorker from 'monaco-editor/esm/vs/language/typescript/ts.worker?worker'
import jsonWorker from 'monaco-editor/esm/vs/language/json/json.worker?worker'
import editorWorker from 'monaco-editor/esm/vs/editor/editor.worker?worker'

It's bundled with the monaco-editor package directly, but with monaco-vscode-api/monaco-languageclient, they are separate packages (@codingame/monaco-vscode-standalone-XXX-language-features and @codingame/monaco-vscode-standalone-XXX-language-features/worker for the... worker)

and an extension such as: https://www.npmjs.com/package/monaco-graphql which assume some monaco contribs are there.

With the new version of monaco-languageclient this seems to not be possible anymore because my understanding is that it doesn't use monaco but vscode instead.

The monaco-editor api is still exposed via the @codingame/monaco-vscode-editor-api package, so anything supposed to work with monaco-editor should work with @codingame/monaco-vscode-editor-api

The only exception is libraries importing internal monaco-editor modules directly, those modules should be whitelisted in monaco-vscode-api (which seems to be the case for monaco-graphql, I'll whitelist them)

How can I support those languages as I did previously ? I see there are vscode language default extensions that could support most of the languages except graphql, but it seems to me that it requires textmate and that it's heavier than the usual monaco language support.

Since a few weeks ago, you can chose to use monaco/monarch languages (If you don't enable theme/textmate services).

The default ones are packaged in @codingame/monaco-vscode-standalone-languages and any package registering a new language should work (in the condition that there is a proper alias from monaco-editor to @codingame/monaco-vscode-editor-api)

[kaisalmen]

Hi @rubenfiszel the following example let's you start the same Langium Grammar DSL example with monarch and textmate grammars: https://github.com/TypeFox/monaco-languageclient/blob/main/packages/examples/src/langium/langium-dsl/wrapperLangium.ts

monaco-editor-wrapper can be run in classic mode (monarch/monaco) or extended mode (textmate/vscode). It removes the need for all the boiler-plate code and you can just configure the languageclient.

Check the Configuration section it details which services are loaded in which configuration.

[kaisalmen]

Also, the multi-editor example makes use of the standalone packages @CGNonofr mentioned above.

[CGNonofr]

With https://github.com/CodinGame/monaco-vscode-api/pull/391, I've managed to make monaco-graphql work

[rubenfiszel]

Thanks for the super quick responses. I will give it a try this weekend with all this new information!

[rubenfiszel]

Ok I've played with it for a few hours and I've hit a few roadblocks.

I am using vite and have different issues between npm run dev and npm run build.

Some issues I now suspect would be solved by using https://github.com/CodinGame/monaco-vscode-api#if-you-use-vite except I import custom wasm in our codebase and we now see new errors:

✘ [ERROR] Cannot find package 'windmill_sql_datatype_parser_wasm_bg.wasm' imported from /git/windmill/frontend/node_modules/windmill-sql-datatype-parser-wasm/windmill_sql_datatype_parser_wasm.js [plugin import.meta.url]

    node_modules/esbuild/lib/main.js:1435:27:
      1435 │         let result = await callback({
           ╵                            ^

    at __node_internal_ (file:///git/windmill/frontend/node_modules/import-meta-resolve/lib/errors.js:431:11)
    at new NodeError (file:///git/windmill/frontend/node_modules/import-meta-resolve/lib/errors.js:374:5)
    at packageResolve (file:///git/windmill/frontend/node_modules/import-meta-resolve/lib/resolve.js:1038:9)
    at moduleResolve (file:///git/windmill/frontend/node_modules/import-meta-resolve/lib/resolve.js:1101:20)
    at defaultResolve (file:///git/windmill/frontend/node_modules/import-meta-resolve/lib/resolve.js:1266:15)
    at resolve (file:///git/windmill/frontend/node_modules/import-meta-resolve/index.js:32:12)
    at file:///git/windmill/frontend/node_modules/@codingame/esbuild-import-meta-url-plugin/dist/esbuildImportMetaUrlPlugin.js:16:34
    at requestCallbacks.on-load (/git/windmill/frontend/node_modules/esbuild/lib/main.js:1435:28)
    at handleRequest (/git/windmill/frontend/node_modules/esbuild/lib/main.js:732:17)
    at handleIncomingPacket (/git/windmill/frontend/node_modules/esbuild/lib/main.js:757:7)
    at Socket.readFromStdout (/git/windmill/frontend/node_modules/esbuild/lib/main.js:680:7)
    at Socket.emit (node:events:519:28)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)
    at Pipe.callbackTrampoline (node:internal/async_hooks:130:17)

  This error came from the "onLoad" callback registered here:

    node_modules/esbuild/lib/main.js:1293:20:
      1293 │       let promise = setup({
           ╵                     ^

    at setup (file:///git/windmill/frontend/node_modules/@codingame/esbuild-import-meta-url-plugin/dist/esbuildImportMetaUrlPlugin.js:8:9)
    at handlePlugins (/git/windmill/frontend/node_modules/esbuild/lib/main.js:1293:21)
    at buildOrContextImpl (/git/windmill/frontend/node_modules/esbuild/lib/main.js:979:5)
    at Object.buildOrContext (/git/windmill/frontend/node_modules/esbuild/lib/main.js:788:5)
    at /git/windmill/frontend/node_modules/esbuild/lib/main.js:2224:68
    at new Promise (<anonymous>)
    at Object.context (/git/windmill/frontend/node_modules/esbuild/lib/main.js:2224:27)
    at Object.context (/git/windmill/frontend/node_modules/esbuild/lib/main.js:2048:58)
    at prepareEsbuildOptimizerRun (file:///git/windmill/frontend/node_modules/vite/dist/node/chunks/dep-whKeNLxG.js:52821:35)

When using npm run build, it is unable to load the workers because it is trying to fetch them at:

http://localhost:3001/node_modules/monaco-editor-wrapper/dist/workers/tsWorker-es.js

which doesn't exist in the dist folders (this is using npm run preview, after npm run build)

[rubenfiszel]

Also it seems graphql doesn't exist as a standalone vscode-api feature package

[kaisalmen]

which doesn't exist in the dist folders (this is using npm run preview, after npm run build)

@rubenfiszel With useWorkerFactory you can set a different basePath (see here) or you can freely redefine how any of the workers should be loaded (see here).

[CGNonofr]

I import custom wasm in our codebase and we now see new errors:

How is it imported in your codebase?

Also it seems graphql doesn't exist as a standalone vscode-api feature package

What do you mean? You can either use the graphql VSCode extension or the monaco library now

[rubenfiszel]

What do you mean? You can either use the graphql VSCode extension or the monaco library now

I didn't find a package @codingame/monaco-vscode-graphql-default-extension to install, whereas I found the ones for powershell, shellscript, go, etc. Maybe I'm missing something but the monaco-editor had one.

How is it imported in your codebase?

Like this:

import init, {
    ...
} from 'windmill-parser-wasm'
import wasmUrl from 'windmill-parser-wasm/windmill_parser_wasm_bg.wasm?url'

init(wasmUrl)

https://github.com/windmill-labs/windmill/blob/863550a91d8380cd18cc750dc63dfa75bdf504bb/frontend/src/lib/infer.ts#L26

Those packages are custom to us (but public) and build by ourselves using wasm-pack: https://rustwasm.github.io/docs/wasm-pack/

[rubenfiszel]

Any clue about why the workers are imported from:

http://localhost:3001/node_modules/monaco-editor-wrapper/dist/workers/tsWorker-es.js

when doing a static build ?

I would have expected vite to export those into an assets folder in the dist/build folder and refer to those directly

[CGNonofr]

I didn't find a package @codingame/monaco-vscode-graphql-default-extension to install, whereas I found the ones for powershell, shellscript, go, etc. Maybe I'm missing something but the monaco-editor had one.

Graphql is not a VSCode default-extension, but you can use the vsix-plugin to load the vsix of the vscode-graphql extension (or by the extension gallery view)

How is it imported in your codebase?

Like this:

import init, {
  ...
} from 'windmill-parser-wasm'
import wasmUrl from 'windmill-parser-wasm/windmill_parser_wasm_bg.wasm?url'

init(wasmUrl)

Maybe esbuild-import-meta-url-plugin conflicts with that way of importing the wasm asset? what if you use the new syntax for it as well?

init(new URL('windmill-parser-wasm/windmill_parser_wasm_bg.wasm', import.meta.url).href)

[rubenfiszel]

Graphql is not a VSCode default-extension, but you can use the vsix-plugin to load the vsix of the vscode-graphql extension (or by the extension gallery view)

What I'm trying to import is:

https://github.com/microsoft/monaco-editor/blob/8503aef01e4f06dff414c15904a1d961e25eb8eb/src/basic-languages/graphql/graphql.ts#L4

so if I understand right it's part of default monaco but not default vscode?

Maybe esbuild-import-meta-url-plugin conflicts with that way of importing the wasm asset? what if you use the new syntax for it as well?

Will try, thanks!

[CGNonofr]

Graphql is not a VSCode default-extension, but you can use the vsix-plugin to load the vsix of the vscode-graphql extension (or by the extension gallery view)

What I'm trying to import is:

https://github.com/microsoft/monaco-editor/blob/8503aef01e4f06dff414c15904a1d961e25eb8eb/src/basic-languages/graphql/graphql.ts#L4

so if I understand right it's part of default monaco but not default vscode?

Oh my bad, you're just talking about the syntax highlighting, not the intellisense, then the package you're looking for is @codingame/monaco-vscode-standalone-languages which includes all monaco basic languages

[kaisalmen]

Any clue about why the workers are imported from:

@rubenfiszel it tries to load the bundled workers that are part of the npm package from a relative location. That is why you can override the basePath. This is especially helpful when you have pure HTML+JS.

The other possibility is to override all required instructions. new Worker(... instructions are then picked up by bundlers like rollup (vite production).

[kaisalmen]

Hi @rubenfiszel any news regarding this?

[rubenfiszel]

I did another attempt 2 weeks ago but was unsuccessful but also didn't have enough time to investigate why.

Our current setup work perfectly which is why it's not urgent to upgrade. All of our code is open-source: https://github.com/windmill-labs/windmill

I was wondering if you provided any consulting services since our WebIDE is a very important part of our product, and monaco-languageclient is the core of it.

As soon as I have some more cycles, I will come back to it. Thanks a lot for the support and detailed answers so far!

[kaisalmen]

Hi @rubenfiszel TypeFox offers consulting if you want more direct or timely support. Please use the contact form on our website to get in direct contact with us.

Independently of that I hope that we now reached a state where future updates will not be as disruptive as they were in the past. We will add more examples/verification examples that ensure directly the stack is working.