Julia syntax highlighting seems very limited

[EssamWisam]

using Imbalance
using DataFrames
using StatsBase

probs = [0.5, 0.2, 0.3]                         
num_rows, num_cont_feats = 100, 5

X, y = generate_imbalanced_data(num_rows, num_cont_feats; 
                                probs, rng=42)                       
StatsBase.countmap(y)

# apply rose
Xover, yover = rose(X, y; s = 0.3, ratios = Dict(0=>1.0, 1=> 0.9, 2=>0.8), rng = 42)

Notice how markdown on Github colors the modules next to using and the function names. This isn't the case with any @example or @repr code I write using documenter.

Is there anyway to improve the syntax highlighting?

[mortenpi]

Documenter uses highlight.js, so this is an upstream problem.

• https://github.com/highlightjs/highlight.js/blob/main/src/languages/julia.js
• https://github.com/highlightjs/highlight.js/blob/main/src/languages/julia-repl.js

One thing you could check is whether Documenter's master version improves this -- we're going from v11.5.1 to v11.8.0.

[EssamWisam]

I hope that it is better on the master's version. The demo from highlight.js goes way beyond just highlighting digits for julia given the example above. They also have support for multiple themes. I think it would super nice if this feature is accessible from here.

I remember back when I started learning Julia, the limited highlighting was something I was immediately not fond of.

[mortenpi]

Do you mind testing with Documenter #master to verify? The issue may also come from the theme or something.

[rbeeli]

@mortenpi Any progress on this? Would you consider using a different syntax highlighting library from highlight.js for Julia, or even in general?

The current syntax highlighting is really a turn off and reduces the quality and reading experience of the documentation, which is IMO very important for the developer experience.

[mortenpi]

Contributions very much welcome, this issue is not on my radar right now.

Would you consider using a different syntax highlighting library from highlight.js for Julia, or even in general?

I'd say so, though we'd need to double check if that choice is somehow tied in the APIs.

But have things improved in highlight.js by any chance? I.e. would bumping its version help? If not, would it be possible to use the highlighter VS Code uses, since I assume the rules for that are being maintained a bit more actively?

Also, a more involved solution would be to build something on top of JuliaSyntaxHighlighting.jl. However, that has the issue that we'd have to drop support for older Julia versions.

[rbeeli]

@mortenpi highlight.js doesn't seem to have improved by much.

I personally think it's the wrong approach to rely on highlight.js, which seems to be far away from the Julia community. We have already solved the syntax highlighting problem with the Julia VSCode extension, which provides excellent syntax highlighting. As you suggested, we should try to go down that road.

Interestingly, it seems there is a JavaScript library that can highlight code based on language grammar files VSCode uses, it even lists Julia as an example, see shiki.style.

Julia + Monaki looks nice and pretty much identical to how it looks in VS Code, same for the GitHub style. It could be used as general replacement for syntax highlighting. It does support ahead-of-time generation of highlighted code in HTML format, no JS required when served.

What do you think about shiki library? Haven't looked at it in detail, but seems promising...

[fredrikekre]

Is it not mostly a style issue? Documenter default style is not so colorful, but the identification of tokens is I think OK?

[rbeeli]

@fredrikekre Tokenization in highlight.js does not seem to be on-par with what is provided in VSCode. VSCode based highlighting has the advantage that it will stay up-do-date with no additional effort, as it's a high-priority project as it seems.

If shiki is an option, I would be willing to try out integrating the CDN version in a fork of this repo, it should be straightforward in the JavaScript mode, which works similarly to highlight.js. Thoughts?

[fredrikekre]

Another alternative is for Documenter to bundle https://fredrikekre.se/posts/highlight-julia/.

I don't really have a strong opinion, but I thought that if we would move away from highlight.js it would be towards something like JuliaSyntaxHighlighter.

[EssamWisam]

I think this is partly a highlight.js version or theme issue. I remember improving the highlighting for DataScienceTutorials.jl by using a newer version (and probably another theme).

Since highlights.js is widely used, I think it would be better to flag and/or solve the issue there but let's at least get Documenter.jl to use the best of it for now.

[rbeeli]

Hi all,

Exciting news! I was able to create a proof-of-concept using shiki.js in place of highlight.js, example see:

https://rbeeli.github.io/Documenter.jl/docs/build/man/guide/

Obviously, the integration code is not production-ready as it is a proof-of-concept, but generally, it looks very good and feels like a breath of fresh air.

The theme used is monokai, which is used for both light and dark mode. Currently best for light mode. It is easily possible to change the theme, or have a different theme for light and dark mode.

Please have a look and let me know your thoughts. I plan on using this in my packages and it would be nice if this could benefit everyone.

[EssamWisam]

Exciting news! I was able to create a proof-of-concept using shiki.js in place of highlight.js, example see

This is extremely beautiful. Would love to see this being added to Documenter (with the option to choose the theme).