Why not use monaco edtior custom languages feature for hightlight?

[Joeoeoe]

Hi, David, your work is very wonderful. 🎉

But I am also curious about why not using custom languages for jsx/tsx hightlight? 🧐

for example in playground: https://microsoft.github.io/monaco-editor/playground.html#extending-language-services-custom-languages

Whth this config, I can add color to tag:

// Register a tokens provider for the language
monaco.languages.setMonarchTokensProvider('mySpecialLanguage', {
    tokenizer: {
        root: [
            [/\[error.*/, 'custom-error'],
            [/\[notice.*/, 'custom-notice'],
            [/\[info.*/, 'custom-info'],
            [/\[[a-zA-Z 0-9:]+\]/, 'custom-date'],
            [/<[^>]+>/, 'tag']
        ]
    }
});

// Define a new theme that contains only rules that match this language
monaco.editor.defineTheme('myCoolTheme', {
    base: 'vs',
    inherit: false,
    rules: [
        { token: 'custom-info', foreground: '808080' },
        { token: 'custom-error', foreground: 'ff0000', fontStyle: 'bold' },
        { token: 'custom-notice', foreground: 'FFA500' },
        { token: 'custom-date', foreground: '008800' },
        { token: 'tag', foreground: '#D34899'},
    ],
    colors: {
        'editor.foreground': '#000000'
    }
});

[luminaxster]

Thank you so much for such kind words, Jonithan!

I considered that approach, and also getting the JSX tokens directly from Monaco's web worker, which was proposed a while back in Monaco's repo. However, that meant I had to hard code the JSX syntax in the former, or wait for the feature request to be implemented in the latter.

I would love the get the JSX tokens directly from Monaco, it does actually have them but they are not exposed via API (last time I checked was January 2021). In that way, we would not have to worry about providing our own tokenizers, and worse, maintaining them. Take, for instance, Fragments <></> seems not to be considered by your tokenizer. Then, add JSX property, element, text, and other elements that may be added.

Interestingly, using Babel to get the AST, simplified what I had to handle, rather than tokens, I identify syntax expressions, which are easy to customize and provide comment actions.

I closed the issue, please feel free to continue the conversation.

[luminaxster]

Thank you for fixing the error in the documentation as well =)

[luminaxster]

You know what? It seems fun to push that path, I'll look into it, any help is welcome. At the time, I didn't have time (and still don't LOL), but we can get it start it.