Portable regex-based syntax highlighting

[dougalm]

A bit of syntax highlighting goes a long way towards making things more readable. Right now we have a little emacs mode and an independent syntax highlighter for the HTML renderings using parser combinators. Does anyone know of a portable solution that lets us define a bunch of regexes and generate modes for vim, emacs, sublime etc?

[paulyoung]

The closest I’ve come across is Iro but that doesn’t support Vim or Emacs.

[danieldjohnson]

Perhaps one could daisy-chain that with something like tm2deftheme or coloration?

[sureyeaah]

As already suggested by @paulyoung and @danieldjohnson, we can use Iro (for VSCode, Eclipse, Intellij, Sublime Code, Atom and a few others), daisy-chain that with tm2deftheme (for emacs) and coloration (for vim and a few others). I'll take this one :)

Edit: tm2deftheme is apparently for themes and not syntaxes, I think.

[sureyeaah]

This is what I came up with for VS Code. I limited it to only check on a token basis instead of trying to mimic the entire parser so that it's flexible as the language's syntax might change. If this looks alright, I'll publish the plugin and try to recreate this for vim.

[apaszke]

That looks great! One minor point would be that we only use upper-case letters for types as a convention, not because it's a requirement (unlike in Haskell), so matching on that can sometimes incorrectly color regular functions such as IToF. But it's no big deal and this is already a huge step forward!

[sureyeaah]

@apaszke Gotcha, I'll publish this so we have something to start with and then have a follow-up PR with more context sensitive highlighting.

• Should I add the plugin to this repo or a separate one?
• Is there something licensing related that I need to keep in mind before pubishing to vscode marketplace?

[joaogui1]

So, what's the state of this issue? It looks pretty useful @sureyeaah @apaszke

[sureyeaah]

@joaogui1 Forgot about this, will send a PR this week.

[apaszke]

@sureyeaah adding stuff to this repo sounds reasonable for now. I don't know anything about the licensing issues of vscode marketplace and I suspect that it depends on what you want to put up there. Self-contained scripts that define syntax highlighters that are entirely authored by you should be ok. Distributing any Haskell components would be more tricky, because we would have to bundle all the licenses of our dependencies.

[joaogui1]

Pinging @sureyeaah

[sureyeaah]

@joaogui1 sorry, I'm working on this right now.

[sharadmv]

For what it's worth, I wrote a simple vim plugin: https://github.com/sharadmv/dex-vim.

Dex code looks like this:

Hope it's helpful!

[sureyeaah]

@sharadmv Thanks! I'll merge this PR and then if it's alright, I'll make some improvements to these plugins and your vim one as well.

[sharadmv]

Yes please! It's my first vim plugin so I don't entirely know what I'm doing.

[general-rishkin]

This is what I came up with for VS Code. I limited it to only check on a token basis instead of trying to mimic the entire parser so that it's flexible as the language's syntax might change. If this looks alright, I'll publish the plugin and try to recreate this for vim.

Is this still active. How can I get a hold of it for use in VSCode?

[sureyeaah]

Need someone to take ownership of this PR :(

[axch]

For the record, PR #503 was an attempt here; reading the conversation there, it was close, but didn't quite make it over the line. Could be picked up to complete this issue, or one could start fresh.

[benjamin-macadam]

Would a tree-sitter grammar make sense? This would be a two birds/one stone situation for me, since I'd also like to get a better handle on how Tree-sitter works.

[apaszke]

Yeah, tree-sitter seems really promising! That would be great.

[youqad]

Oops, I just saw this! For what it’s worth, as I couldn’t find a syntax highlighting extension for Dex in the Marketplace, I wrote a little VS Code extension based the regexes of the Emacs Lisp file: https://marketplace.visualstudio.com/items?itemName=youqad.dex-syntax-highlighting

[richardwesthaver]

any luck with a tree-sitter-dex implementation? I would be interested in working on that but not sure how it aligns with some of the current project goals (https://github.com/google-research/dex-lang/issues/1304) - may not be a priority for the project if dex web is the IDE and the LSP to support it is WIP. I would love to add dex to my list of tree-sitter parsers though :)