Syntax highlight for symbols

[hayd]

It would be great if symbols (e.g. :hello) were hightlighted, e.g. like strings are.

[carlobaldassi]

It's extremely hard to do this right without the full Julia parser, because of the overloading of the : in julia, which is used for ranges, symbols, ternary operators (plus type annotations such as :: and <:). I did implement the recognition of some special cases out of necessity, but it's far from perfect. I'll have a look if the approach can be extended in a reasonable way.

[hayd]

Thanks! I posted here is after reading that it's coming to emacs (at least in simple cases), hopefully the following links might be of some use: http://www.wilfred.me.uk/blog/2014/11/16/editing-julia-code-with-emacs/ https://github.com/JuliaLang/julia/pull/9024

[Wilfred]

FWIW, the Emacs mode is currently using a simple regex, rather than doing any clever parsing: https://github.com/Wilfred/julia/blob/cf1f9af03b19f7d3d2598ae476ce16d21d5cbf00/contrib/julia-mode.el#L205 (hopefully that regexp is readable to you, we're requiring `,[or(before the:`)

This is sufficient to pass all the cases in my test file and was sufficient on the real-world code I tried it with.

[ararslan]

It may be possible to mimic how the Vim syntax file for Ruby does it, since IIRC Ruby also uses : for a number of different things, including symbols. Though I do think Julia's use of : is more complicated than Ruby's.

[ExpandingMan]

Not that I know what I'm talking about, but would looking at the code for the Atom implementation ("Juno") be helpful? That supports proper highlighting for Symbols. As Symbols occur one hell of a lot in Julia, it would be really, really nice to have this feature and it would make code much easier to look at.

The absurd abuse of the ":" symbol in Julia is the one thing I don't like about the language. It looks especially terrible in the context of dataframes where Symbols and ranges are used in the same context. I considered raising an issue on the Julia's repo suggesting allowing & to be used as the character for specifying Symbols with the intention eventually deprecating :, but I could just imagine the things people would say to me.

[carlobaldassi]

I have finally finished an implementation of this. Before I merge it into master, can someone stress-test it and tell me if you find any regressions/bugs? It's in the hlsymbols branch.

[ararslan]

@carlobaldassi I'll try it out and let you know if I run into anything weird. Looking forward to this!

Edit: Looking rock-solid so far. 👍

[ExpandingMan]

I'll also try it out soon. Thanks for working on this! This might sound silly but :s in general are my one big aesthetic hangup about Julia, and lacking the text highlighting for it was really making me feel it... Now I'll be able to feel so much joy when looking at my code!

[ararslan]

I've thrown just about everything I can think of at it, and so far everything looks 💯. The only somewhat weird case is :1, which is highlighted as a symbol but isn't technically a symbol. Shouldn't matter though, since no one should be writing :1 anyway. IMO this change is production ready.

[MichaelHatherly]

Thanks @carlobaldassi, looks much nicer with symbols highlighted! Haven't noticed any highlighting bugs yet, though with a large number of symbols per page, such as in https://github.com/JuliaDocs/Highlights.jl/blob/master/src/lexers/julia.jl, all movements start to lag significantly to the point of not being usable.

[carlobaldassi]

Thanks. The lag is incredibly annoying and frustrating, especially because it is not entirely deterministic. It is therefore very hard to test. For example, I tried to simplify some regexes to see if it solved the problem, and it did, except sometimes (on the same file, opening it and starting scrolling works fine say 90% of the times, and 10% of the times it becomes slow or even unresponsive). This may seem to indicate a vim bug. I'll need to keep investigating to see if there is anything that can be done about it.

[carlobaldassi]

I have updated the branch. It's faster now, although I had to give up the recognition of ranges of the form a :b and of quoted numeric literals.

I'm not sure that we aren't still triggering some vim bug. Could you guys try again to see if it is usable/stable/correct?

[ExpandingMan]

I've only opened files with large numbers of symbols since your latest commit, and so far it seems fine. Looking at htop I do not see any issues with unreasonable CPU utilization.

I think giving up recognition of a :b as a range is more than fine; frankly I think that syntax should be disallowed anyway as it is incredibly confusing to look at. I'm not entirely sure what you mean by "quoted" numeric literals... :(123) and quote 123 end work fine (i.e. the numerals appear in red).

Thanks for the fix.

[MichaelHatherly]

Yes, seems better now as far as I can tell, some very minor lagging when moving one line at a time up and down fast, but not particularly noticeable, so I'm happy with it.

[ararslan]

I'm not sure that we aren't still triggering some vim bug.

FWIW I use NeoVim rather than Vim. Assuming the other folks who've tested did so on Vim, if this the lag is due to an upstream bug, the bug must present in both Vim and NeoVim. While not impossible, I think that makes it somewhat less likely that it's an upstream bug.

I had to give up the recognition of ranges of the form a :b

I was actually amazed in my initial testing that you even bothered to catch that case. It's likely uncommon in the wild, so probably fine.

and of quoted numeric literals.

If by that you mean :1, for example, I think that's totally fine, since those aren't actually symbols anyway. typeof(:1) -> Int64.

[carlobaldassi]

Yes, with "quoted numeric literals" I meant things like :1. They are actually quoted, but it's subtle:

julia> typeof(:1)
Int64

julia> :(:1)
:($(QuoteNode(1)))

Anyway, since nobody actually writes :1, quoting them was mostly intended as a way to catch the case of unintended quoting (i.e. expecting a range and getting a quoted literal instead). It's true that it is no big deal.

For the case of ranges written as a :b, it's true that nobody should do that (and it would make sense to disallow it), but still that's the way Julia parses them, so I just wish we could be faithful to the parsing. But it's better overall to have false positives than false negatives when detecting symbols, so again we'll just live with that tradeoff I suppose.

As for vim vs neovim, I think it's unlikely that neovim has touched the original vim regex engine or the syntax highlighting, so I wouldn't be surprised if there was a bug in both. I don't think that what I had observed (the editor non-deteministically becoming unresponsive when scrolling) could be explained in another way than a problem with that. I also had other issues anyway even previously, like often having to hit CTRL-L to refresh the highlighting.

I'll wait a little bit more to see if any of us catches some other issue "in production"; barring that I'll merge in a few days. Thanks to everybody for the feedback.

[ExpandingMan]

Yeah, if it were up to me, syntax like :1 would be forbidden anyway so, again, I'm perfectly happy with it not being recognized.

Anyway, I've been using this pretty extensively for the last couple of days, and for the most part it's looking pretty good. I definitely think there is some vim bug, however: sometimes I'll get sporadic performance problems, though they are not unbearable. As far as I can tell, there is no rhyme or reason to them whatsoever, so I suspect it will be quite difficult to pin down whether it's a vim bug or a problem with the julia-vim code. It may also have something to do with you-complete-me (which frankly I have long found generally somewhat buggy).

The problems are limited enough that I'll definitely put up with them to get symbol highlighting! Thanks again.

[carlobaldassi]

Update: I was still experiencing those very annoying slowdown problems occasionally, so I tweaked things. It seems to have miraculously worked, and I have also reinstated the recognition of ranges such as a :b and of quoted numerals (this is in the last commit, it can be peeled off if it proves too much again). I even measured the scrolling time on the particularly nasty file that I had been using in testing, and it is the same as without the plugin (so no slowdown whatsoever, just a little flickering of the line numbers at most).

I'll still try and use this version for a few more days; of course, everyone else is also encouraged to pull from the hlsymbols branch again and test/report issues as well.

[MichaelHatherly]

I've been using this for most of today without any problems. :100:

[ExpandingMan]

Looks quite good, as far as I can tell the performance issues are fixed.

One small problem I notice is that in the line return :symbol the symbol is not highlighted, presumably because it is being recognized as a range. Personally I would gladly take reliable symbol highlighting over the ability to recognize the horrible range syntax a :b. Keep in mind that return is still necessary in Julia if one is doing a conditional return before the end of a block.

Anyway, thanks again for your work on this, it really is a huge relief to have this feature.

[carlobaldassi]

Another day, another update. It turned out that the performance improvement was not a miracle but rather an embarrassing bug (it was recognizing only a subset of all possible Unicode operators). Still, I moved from there and have fixed the problem now, while the performance is still good, at a reasonable — I think — compromise in parsing accuracy.

The code in the branch now also recognizes quoted expressions such as :(expr). With this, it seems to be able to parse correctly everything in JuliaParser's parser.jl and lexer.jl (the latter is the only case in which I could observe a slight scrolling slowdown due to the density of Unicode operators and quoting symbols).

@ExpandingMan: nice catch about the return :symbol case, I fixed it. About the a :b ranges, I came up with one possible example where it may make sense to use it (and expect it to be properly highlighted), in code like the following;

for r in [(a  :b  ),
          (a+2:b+2),
          (a-2:b-2)]
    # do something
end

[ExpandingMan]

Looking great.

There's just one more thing about the highlighting as it is now that bothers me slightly: the character ; isn't colored. I feel it probably should be colored just like : and keywords, as it is a perfectly good special character.

Thanks again.

[carlobaldassi]

Ok it's in master.

@ExpandingMan I have added the possibility to highlight semicolons if you want, you need to do something like this in your .vimrc:

hi link juliaSemicolon Operator

(Or use whatever you like instead of Operator, just use :hi to see what's available.) (BTW there was always the possibility to do this with parentheses as well, with :hi link juliaParDelim Delimiter. I thought I'd mention it since it's not documented.)

I'm not turning it on by default since to me a semicolon is punctuation, not an operator, and I've never seen punctuation highlighted.

[ExpandingMan]

Hmm, I guess you're right about the semicolon. I think of it as the same as quote and end, which are highlighted, it didn't really make sense for me for end to be highlighted and ; not to be. Also, it's nice to see the highlighting if you have multiple semicolon separated statements on one line, it helps my brain separate them.

Anyway, thanks for adding the functionality!