Semantic highlighting with monaco-editor options

[kaisalmen]

Hi @CGNonofr as you may have noticed I moved over the Langium gammar language client and server from monaco-languageclient to monaco-components:

https://github.com/TypeFox/monaco-components/blob/main/packages/examples/src/langium/wrapperLangium.ts

The monaco-editor-wrapper allows to run monaco-editor with monaco-languageclient in a classic (regular monaco-editor config) or as extension like configuration. The ported example allows to run the editor in both modes. That generally works, but semantic highlighting only works in extension mode (see green Abstract type def in screenshots below). If textmate / themes services are not configured, I expect that to work. Do you agree? Or have I overlooked something here?

Classic monaco-editor configuration:

Extension driven configuration:

Side question: Is it possible to de-initialize all services? Currently you can dispose the editor and language client, but when the service init is done once it cannot be disposed or revoked. Is this even possible?

Easiest way to reproduce is cloning the repo, then npm i && npm run build and afterwards npm run dev; optionally run npm run watch in parallel if lib code needs adjustment.

[CGNonofr]

The monaco-editor-wrapper allows to run monaco-editor with monaco-languageclient in a classic (regular monaco-editor config) or as extension like configuration. The ported example allows to run the editor in both modes. That generally works, but semantic highlighting only works in extension mode (see green Abstract type def in screenshots below). If textmate / themes services are not configured, I expect that to work. Do you agree? Or have I overlooked something here?

By semantic highlighting, do you mean syntax highlighting? If so, monaco features are disabled by the treemending patch to not conflict. If you want the monarch highlightling back, you can import monaco-editor/esm/vs/basic-languages/monaco.contribution.js

Semantic won't work with monarch I think, we had a hack back then to make it work

Side question: Is it possible to de-initialize all services? Currently you can dispose the editor and language client, but when the service init is done once it cannot be disposed or revoked. Is this even possible?

I wish the answer to be yes, but it's much more complicated

Services are just stored in a service collection (inside StandaloneServices) and used to create new editors. Even if StandaloneServices doesn't currently support re-initializing the services, we would easily be able to make it possible.

The main issue it that some services are registering handlers (snippets, extension points, terminal backend...), not mentioning workers, and the code often doesn't support replacing the handler, so the next service instantiation would fail.

So it's probably hackable, but it would be very complicated.

Our best chance is probably to run everything in a sandbox-like (new Function or eval?) but I've never tried anything related yet.

The simpler solution is probably just to... reload the page. That what vscode does it everytime you change your workspace

[kaisalmen]

If you want the monarch highlightling back, you can import monaco-editor/esm/vs/basic-languages/monaco.contribution.js

The monarch based syntax highlighting works without this specific import. I use the standard monaco-editor since you introduced it quite recently (1.80.x, right?)

Semantic won't work with monarch I think, we had a hack back then to make it work

I meant semantic highlighting and the Langium Language Server provides that (see second screenshot ⬆️). It was working with an older version of the whole lib stack (wrapper 1.6.1 | mlc 4.0.3 | api 1.69.13 | monaco 0.34.1, see https://github.com/TypeFox/monaco-components/issues/32)

I wish the answer to be yes, but it's much more complicated ...

Nevermind, let's not go down this rabbit hole. 🙂

[CGNonofr]

The monarch based syntax highlighting works without this specific import. I use the standard monaco-editor since you introduced it quite recently (1.80.x, right?)

I'm surprised, that import is removed from monaco-editor by the treemending patch

I meant semantic highlighting and the Langium Language Server provides that (see second screenshot ⬆️). It was working with an older version of the whole lib stack (wrapper 1.6.1 | mlc 4.0.3 | api 1.69.13 | monaco 0.34.1, see https://github.com/TypeFox/monaco-components/issues/32)

Yes we had a hack changing the implementation of getTokenStyleMetadata

see https://github.com/microsoft/monaco-editor/issues/1833

[kaisalmen]

Yes we had a hack changing the implementation of getTokenStyleMetadata

see https://github.com/microsoft/monaco-editor/issues/1833

@CGNonofr please help me to translate 🙂

This https://github.com/microsoft/monaco-editor/issues/1833#issuecomment-774156946 means that the support in monaco-editor is not complete (still two years later), meaning full support for semantic tokens is only available in vscode and therefore in this context only if you use the textmate+theme service, correct?

[kaisalmen]

My interpretation: Semantic tokens usually are provided by language servers and with pure monaco-editor this is not available => no priority to support this in monaco-editor. Does that make sense?

[CGNonofr]

What I understand is that it's not supported by the default monaco themes, you need to define a custom theme with additional rules.

But I think it's pretty similar to vscode (the Light and Dark themes don't support semantic tokens) except the default vscode themes (Dark+ and Light+) are based on those and add semantic token colors, so we just need to do the same with monarch themes? (see https://github.com/microsoft/vscode/blob/main/extensions/theme-defaults/themes/dark_plus.json)

Back then when we used the hack, monaco didn't support it at all (getTokenStyleMetadata was empty)

[kaisalmen]

@CGNonofr thank you for the hints. I added a custom theme and I can see that it is applied when I change the base. Interestingly monaco.editor.IStandaloneThemeData does not offer semanticHighlighting nor semanticTokenColors like the vscode themes do.

Using textmate / theme service: Setting "semanticHighlighting": true in a vscode theme (even the default ones have it, btw) makes then language server perform semantic token handling. If I set "semanticHighlighting": false in the vscode theme the semantic token handling code in the LS is not executed. I verified this with debugging.! Something internally happens instructing (capabilities?) the LS to not perform semantic token handling.

Using editor options and monarch + theme: Adding "semanticHighlighting": true or 'semanticHighlighting.enabled': 'configuredByTheme to the theme has no effect, also as well as passing 'semanticHighlighting.enabled': 'configuredByTheme' to the editor options (or both).

Any idea?

[CGNonofr]

For some reasons, the standalone theme service declare not supporting semanticHighlighting (see https://github.com/microsoft/vscode/blob/bb389f28ea44f4a9fad87dbf8c5e0ad7874acc0c/src/vs/editor/standalone/browser/standaloneThemeService.ts#L180)

settings semanticHighlighting.enabled to configuredByTheme will use that flag and being disabled

you'll need to set semanticHighlighting.enabled to true

(Or hack StandaloneTheme to change its semanticHighlighting property to true)

[kaisalmen]

you'll need to set semanticHighlighting.enabled to true

Tried that already, doesn't work. Can't we patch monaco-editor here to treat this properly? 😆

(Or hack StandaloneTheme to change its semanticHighlighting property to true)

will try this now

[kaisalmen]

@CGNonofr When I change this.semanticHighlighting = true in StandaloneThemeService.js, then the LS does semantic token handling.

FYI, regarding vscode themes, the extra colors are defined only in the + and Modern themes, but the flag "semanticHighlighting": true is set everywhere (also default light/dark).

Question still is, why does this have no effect?

createConfiguredEditor(<HTMLElement>, {
    'semanticHighlighting.enabled': true
});

Edit: Corrected copy-paste-errors.

[CGNonofr]

Question still is, why does this have no effect?

createConfiguredEditor(<HTMLElement>, {
    'semanticHighlighting.enabled': true
});

The semanticHighlighting.enabled setting is not used by the editor itself. It need to be registered in the configuration service.

If you use the StandaloneConfigurationService, the extra configuration provided to the editor is registered in the configuration service. If you used the configuration service override, it's not, and you need to either call updateUserConfiguration with it or call StandaloneServices.get(IConfigurationService).updateValue('editor.semanticHighlighting.enabled', true)

[kaisalmen]

@CGNonofr 🎉 YES, using updateUserConfiguration works and it is logical, too in the end. I use model service and config service always in the wrapper, because they are required under the hood.

Ok, I need to expand the wrapper configuration/handling to make this bullet-proof. Thank you very much. Layer 8 issue in the end once more. 😆

[CGNonofr]

I can't blame it on you, it's very complicated ^^