Investigate the use of shiki as the library for syntax highlighting

[Nova38]

Proposal

Use shiki, https://shiki.style/, as the syntax highlighting library for mystmd. It uses the same syntax highlighting engine as vscode and has very good tooling.

• It uses TextMate grammars for its language definitions. Which is the same definitions that are used by vscode and GitHub's linquist among others.
  • The output it produces is very good.
  • there is a large number of high-quality language definitions already defined.
  • It also allows reusing vscode color themes.
• It is supports being run in a compilation step for static generation and at runtime in the end users browser.
• It has a extensive API for defining transformations and decorations

Additional notes

It is used in a lot of open source projects

• Astro
• Nuxt
• vitepress

[agoose77]

I recall seeing this mentioned somewhere ... turns out it was you! https://discord.com/channels/1083088970059096114/1083088970059096117/1295937103846314004

[Nova38]

Yep! I finally have some free time to work on some contributions to a few projects, so I wanted to open the floor to see if this is something that would be desired.

I recall seeing this mentioned somewhere ... turns out it was you! https://discord.com/channels/1083088970059096114/1083088970059096117/1295937103846314004

I also forgot to add that it is really easy to add custom languages to shiki as the consumer of the library. I think that might be useful if someone wanted to have syntax highlight for a custom languages (such as if it was a book about language design/compilers and they made up a simple language) (See https://shiki.style/guide/load-lang)

[choldgraf]

Thanks for this idea @Nova38 ! I wanted to share a few quick questions to help us triage this idea a bit. I promise they're not coming from a place of skepticism - just trying to understand cost/benefit!

• What's the benefit of using this over the current system we use for syntax highlighting? (in functionality, maintenance cost, long-term outlook, UX, etc?)
• How much work would it be to implement and then maintain this?
• Why should this be the most important thing to use maintainer time implementing/reviewing/etc?

[Nova38]

No worries, I completely understand. Those are excellent questions! I will work on writing up those and should have them in a short bit.

To aid the comparison is there a specific package that handles the code directive/syntax highlighting directive? I used the GitHub search to find where https://github.com/react-syntax-highlighter/react-syntax-highlighter was being used, but I am unfamiliar with the codebase and want to make sure I am comparing it to the correct spot. If you know of the top of your head could you point me in the right direction please? If not don't worry about it I'll just keep going through it.

Thanks!

[rowanc1]

The syntax highlighter is used in the myst-theme package here:

https://github.com/jupyter-book/myst-theme/blob/main/packages/myst-to-react/src/code.tsx#L106

[Nova38]

Here are some of my initial thoughts

• What's the benefit of using this over the current system we use for syntax highlighting? (in functionality, maintenance cost, long-term outlook, UX, etc?)

  • An example of what is possible, here is it intigrated with typescripts Shiki Twoslash: Static Code Samples for JS Projects - https://shikijs.github.io/twoslash/

  • The UX of adding new transfomations is a lot more consise

  • Functionality:

    • Uses .tmlanguage definitions, which has more languages that prism/highlight and its definitions are better.
    • Allows access to preform transforms using hast https://github.com/syntax-tree/hast
    • It provides a cleaner api than the current one for adding decorations Decorations | Shiki - https://shiki.style/guide/decorations

      import { codeToHtml } from 'shiki'
      const html = await codeToHtml(code, {
      theme: 'vitesse-light',
      lang: 'ts',
      decorations: [ 
      {
      // line and character are 0-indexed
      start: { line: 1, character: 0 },
      end: { line: 1, character: 11 },
      properties: { class: 'highlighted-word' }
      }
      ]

  • It has some built in transformations that could improve code readability such as the @shikijs/colorized-brackets | Shiki - https://shiki.style/packages/colorized-brackets and Shiki Magic Move Demo - https://shiki-magic-move.netlify.app/ Usng

  • Longterm outlook

    • Language definitions are likely to keep being developed that are compatible as it is used by a lot of tools
    • Built on top of some of the libraries used by vscode (oniguruma)
    • Integrated in to multiple large projects/frameworks (Nuxt, Astro, vitepress, etc... )'
    • They have a rough roadmap Future | Shiki - https://shiki.style/guide/future
    • Some of the core developers also are core developers on things like vitest, nuxt, etc...

  • Maintenance cost

    • Shiki has a diffrent way of parsing what they call as 'meta' strings for customizing the front ends.
      • @shikijs/transformers | Shiki - https://shiki.style/packages/transformers
      • // [!code highlight]
      • We could still use the css and other parts of their transform and just add the classes in the transform that we get from the myst directive props.
      • I am pretty sure that any of these have to be manually enabled so we don't want them to work it should be simple to just not add them

• How much work would it be to implement and then maintain this?

  • Honestly I am not sure how long it would take to implement. It doesn't look like it would take much to add it. To maintain it after the initial setup would probably be minimal. I can make a POC to see what the code will look like. I Think that it might be less work maintaining the current code of the code highlighter.

• Why should this be the most important thing to use maintainer time implementing/reviewing/etc?

  • It could be used to further expand what mystmd can do with code annotations

[Nova38]

Is there a way to override an existing component and inject different project specific component?

[stevejpurves]

@Nova38 no not on the renderer/theme/react side of things, this is part of the plugin framework that has yet to be fleshed out.

+1 for shikijs being pure ESM though. We've had to wrangle with react-syntax-highlighter at times due to the way it is packaged, trying to be a cjs/esm distribution.

[Nova38]

I am not sure that I entirely follow the question. Are you wanting how it by itself can be adapted into a plugin itself, how it could fit in by themselves exposing parts of it that other plugins could then be able to expand on it and add new features to the code blocks, or is it something else?