A few questions

[IlanCosman]

Hi @kopischke 👋 It's great to hear you're working on a Fish extension for Nova. I have a few question:

1. Are you/have you considered making a language server that could be used in other editors as well (I use VSCode)? It would be a shame to duplicate effort.
2. From what I've read, it seems you're doing a more traditional parsing style with a syntax tree etc. (I don't know the terms). I recently though of using fish_indent --dump-parse-tree for this task. The benefits are (1) that we don't have to create our own parser and (2) that it's perfectly integrated with Fish, i.e the extension would always operate as Fish does.

Thoughts? Thanks for working on this 😄

[kopischke]

Hey @IlanCosman, thanks for reaching out. Currently, I am trying to get Nova to highlight fish syntax correctly (less easy than it sounds). Nova’s syntax highlighting system is basically a pushdown automaton based on regex matches. Thanks to a few clever tricks and the use of PCRE2, it is rather powerful, and it has the nice bonus of offering symbolication of constructs for navigation and completion purposes à la ctags, but it is entirely statical. Syntax highlighting via a language server is not supported (is it in any editor? I know it is technically part of the LSP spec, but I have yet to see an implementation).

A language server would of course be great, once I have the basics going. I wasn’t aware of fish_indent’s ability to dump a parse tree; that opens up some interesting prospects, but I am unsure when, if at all, I could look into that.

[IlanCosman]

I think I used the wrong term. Syntax highlighting is also done through regex in VSCode, but Semantic highlighting is provided by a language server, and VSCode supports it.

I wouldn't wish highlighting a language in regex on my worst enemy 😄 https://github.com/bmalehorn/vscode-fish does it this way, but it fails on some edge cases. For example:

I had started a bit of work on a language server for fish, but I was too intimidated by learning Typescript and a new API 😄 I might try again at some point.

Anyways, if Nova supports semantic highlighting through language servers, I think using fish_indent --dump-parse-tree and a language server is clearly the best way to go.

[kopischke]

Oh, I only just noticed the “duplicate effort” part. Are you working on an LSP server for fish already?

[IlanCosman]

Another example of failing regex:

I'm not doubting your regex abilites, but you're human, and therefore you'll mess up and miss some edge cases. Fish itself never messes up, because Fish always does exactly what Fish does 😂

Oh, I only just noticed the “duplicate effort” part. Are you working on an LSP server for fish already?

I've been noodling around a bit, but nothing substantial yet. But https://github.com/bmalehorn/vscode-fish is a perfect example of duplicated effort.

[IlanCosman]

On cursory glance it seems Nova does support highlighting through a language server: https://docs.nova.app/api-reference/language-client/#language-features

[kopischke]

On cursory glance it seems Nova does support highlighting through a language server.

AFAIK, the documentHighlight interface is not intended, nor appropriate, for general purpose syntax highlighting (but I might be wrong; I’m far cry from an LSP expert).

I wouldn't wish highlighting a language in regex on my worst enemy.

Yeah, regex matching is a pain, especially so when the state machine it triggers is pretty simplistic (I’d have preferred Panic, Nova’s makers, had used treesitter for their highlighting; but then, I‘d also preferred they used Node for the extension runtime, and oh, can I wish for a pony too while we are at it 😅?). That is exactly why I have been working on this for going on two months now, and have a 2800+ lines XML syntax definition to show for my pains. It’s highlighting all of your examples correctly, BTW; catching stuff like line continuations, empty expansions and the positional and contextual semantics of fish is exactly what makes the whole endeavour “less easy than it sounds“ 😇.

But you are right, of course, that statical syntax highlighting has serious limitations, starting with the fact that all that is dynamic is out of scope, and working your way down from that to regex complexity considerations and limitations of the state model the syntax highlighting system offers.

you're human, and therefore you'll mess up and miss some edge cases.

Yeah, that too, I am sure. I agree with you developping an LSP server to go beyond that is the right call, but that is a middle to long term project, and I’d like my fish scripts highlighted and completed in Nova about right now 😉.

I was too intimidated by learning Typescript

Hah, I hate TS with a fervour. Have you ever looked at the transpiled code it produces? JS that looks like C, except for the MS provided core – that looks like Assembler. Depending on where you come from, that might even sound good, but my background is in Ruby and Crockford JS, not CompSci C, and that makes me shudder.

[kopischke]

Just FYI, @IlanCosman, I have had a peek at the parse tree dump fish_indent produces because I thought that, short of developing a full-fledged LSP server, it might be useful for error detection.

I have to say I seriously doubt fish_indent can be used for any form of semantic highlighting in its current state. The primary problem is that it’s missing a machine readable mapping to the source it parses, and you cannot easily match back its output because it discards empty tokens. Even if that was resolved, the output misses too many details for general LSP usage as the generated AST basically bails at the “argument” level. Among other things, mapping used variables to their declaration is not supported (variable expansion is just another argument) and significant details are lost (for instance, home expansion or wildcards). The more the pity.

[IlanCosman]

I have to say I seriously doubt fish_indent can be used for any form of semantic highlighting in its current state. The primary problem is that it’s missing a machine readable mapping to the source it parses, and you cannot easily match back its output because it discards empty tokens.

Could you explain this more? It's going a bit over my head because I've never done anything like this 😄

Even if that was resolved, the output misses too many details for general LSP usage as the generated AST basically bails at the “argument” level. Among other things, mapping used variables to their declaration is not supported (variable expansion is just another argument) and significant details are lost (for instance, home expansion or wildcards).

Indeed. I agree that --dump-parse-tree isn't suitable for those reasons you mentioned. My new plan is to use --ansi which doesn't have the problems you mentioned. It recognizes variables/home-expansion/wildcards with different escape codes, which we can parse.

Mapping variables to their declaration will never happen because Fish is a shell language. There's universal, global, and exported vars. So that was always going to be a missing feature.

[kopischke]

Sorry for the late answer – RL has a nasty way of catching up to you in the midst of the most interesting discussions. The issue I see is that, if the AST output of fish_indent is easily matched to the underlying code by a human, that is not true for a machine, because a) there is no positional relationship in the output (not line and column data), and b) the tokens it references are partially expanded, which means they do not necessarily match the input anymore (for instance, echo ""'bar'\ bar would be referenced as string bar bar – contrived example for the sake of the explanation). That makes it impossible to match the AST output to the actual code it is constructed from for the sake of editing, where, unlike in parsing for shell, we need to map to the literal original.

[kopischke]

Closing as there is no actionable issue here (at least for the time being).

[IlanCosman]

Any update on this project? No pressure 😄 I plan to try out Nova when this releases ❤️

[kopischke]

Hey Ilan, sorry for the delay. Partly, RL has been keeping me from tinkering with this but mostly, I have been waiting for the final call on fish’s #7980. Ands a good thing that was, apparently, as it looks like 555af37 is going to be reverted …

EDIT: as to perspective, the idea is to release on the heels of fish 3.3. I have one major undertaking in the syntax file to tackle and a handful of kinks to iron out, so this may not be a “same day” release, but it shouldn’t take much longer.

[kopischke]

For anyone interested in this extension: syntax highlighting for Fish 3.3 is done, but I have run into a Nova bug that cripples completions and am awaiting resolution before publishing.