Add callback to adapt the scope reduction ?

[paul-marechal]

In the tokenizer tokenize method, you convert an array of scopes to just the last scope.

In our case, this is good enough for 99% of the cases, but it fails on punctuation.

Would it be possible to add a callback on this last bit of code, where you return the result of the tokenization, so that we could make our own conversion ? Maybe just use a default callback that does what you are currently doing, but leave use the choice to change it :)

Maybe just add a callback on this last piece:

                        return {
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens.map(token => ({
                                ...token,
                                // TODO: At the moment, monaco-editor doesn't seem to accept array of scopes
                                scopes: token.scopes[token.scopes.length - 1]
                            })),
                        }

Like:

                        return callback({
                            endState: new TokenizerState(res.ruleStack),
                            tokens: res.tokens
                        })

This would allow us to return the last scope except punctuation, for instance. Or maybe format the scopes in the way monaco is expecting it (as it seems that vscode supports scope arrays, using the Developer: Inspect TM Scopes command). I have no idea how it works yet though.

[zikaari]

My implementation might not suit your needs so I'd suggest a PR where you get to express things as you see fit 🙂

---

PS: Something you should keep in mind when using this package is that it's hard to get it working with "stock" monaco-editor distribution. I recommend that you consider alexandrudima's recommnedation:

One cleaner solution would be to create a monaco-editor distribution which does not include monaco-languages (the basic colorization for several languages). This can be easily done by cloning this repo, running npm install, commenting out the relevant lines in /metadata.js, and running npm run release. The /release/ folder will have a monaco-editor without monaco-languages.   Read full comment

[zikaari]

~Also you might consider opening a new issue in monaco-editor repo with a proposal to add support for accepting array of scope names. Seems like it should've been this way from day one.~

Done

[paul-marechal]

Something you should keep in mind when using this package is that it's hard to get it working with "stock" monaco-editor distribution.

I think we are cherry picking the languages because we rely on monaco-editor-core and not monaco-editor.

But I just came to realize that this issue is a bit silly, my bad. I can easily create my own tokenizer using the TokenizerState from your package to define the logic I need.

Basically the problem I had was the fact that you converted an array of scopes to a unique scope. Turns out that the last element of the array is not the best one in my case (I need to ignore punctuation scopes, should do the trick for now).

I saw your issue on vscode to support arrays of scopes, but as a temporary solution as can fix it for my case :)