EncodedTokensProvider?

[vasiltabakov]

Do you plan supporting EncodedTokensProvider?

[zikaari]

What is your use case?

[vasiltabakov]

I'm still researching what exactly the encoded token provider strategy is about. So far it seems to me that using encoded tokens could lead to faster processing of tokens in the editor. So my goal is to find out if using encoded tokens would lead to a faster editor experience.

I've tried implementing it, but it didn't work. The grammars did return tokens in binary format but they didn't seem correct. One of the visible results was that the colorization in the editor was all gone.

So I thought asking, maybe you would know more since this is a textmate thing related to tokenization and I'm not too familiar with either.

[zikaari]

Encoded tokens only work when tokenizer is given a theme object, after which the color values are returned directly instead of token names ( or scopes).

In case of monaco editor, one we don't have access to the theme, and two it doesn't accept color values directly but instead semantic token/scope names (and that too is very hacky)

[vasiltabakov]

In monaco you can define your own theme passing it an IStandaloneThemeData object, would that be enough? I tried following some of the code of vscode and so implementing encoded tokenization won't be straightforward. I'll update the issue if I find a working solution.

[bolinfest]

@NeekSandhu @vasiltabakov

I just got a demo working that uses vscode-textmate and vscode-oniguruma. It leverages EncodedTokensProvider:

https://github.com/bolinfest/monaco-tm/