Use shiki for code highlight

[felipecrs]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

Motivation

Have you guys ever considered switching to Shiki for code highlight, instead of Prism?

I find Shiki a lot nicer, with much better grammars that are able to highlight much more code than Prism does.

Also, Shiki is being actively maintained, while Prism seems a little bit abandoned. However, even if Shiki wasn't being maintained, its themes would never stop improving since they are taken from VS Code anyway.

Here is a quick comparison I did:

Prism (dracula - the default theme from Docusaurus):

Shiki (dracula):

PS: I'm aware of docusaurus-preset-shiki-twoslash, and that's what I used for the example above (https://github.com/felipecrs/shiki-docusaurus-test). This is to suggest a better default for the project as a whole.

Maybe it's a good breaking change candidate for Docusaurus v3?

Self-service

• [ ] I'd be willing to do some initial work on this proposal myself.

[Josh-Cena]

Several things, based on my experience with Shiki, do correct me if I'm wrong:

1. Shiki is server-side and generates static markup during Markdown parsing (using a Rehype plugin, IIRC), so there's no way to use it on a React page.
2. Shiki doesn't support as many languages as Prism, neither is it as easy to add them.
3. Shiki being purely server-side means it cannot be used in live code blocks, so there's inconsistency out-of-the-box.

What do you think?

[felipecrs]

I'm actually confused.

PS: I know nothing about frontend nor React.

1. But looking at Shiki's documentation, I don't see any indicator that it runs on server-side. In fact, it can run on server-side with Node.js. But it also runs in the browser, including a WebAssembly library to perform language tokenization.

   About using with React, I believe this may help: https://www.npmjs.com/package/remark-shiki-twoslash

2. I think that's debatable. If there is a text grammar compatible with VS Code (and several other editors which uses the same standard), it's easy to port to Shiki. For example, I use a lot of Jenkinsfiles, and Prism's support for it is limited to Groovy. There are a few dark VSCode extensions that adds rich syntax highlight for declarative Jenkins pipelines that I could then easily port to my website. I have a feeling that the list of languages supported by Shiki tends to increase more than Prism does, also, maybe the languages not supported by Shiki but supported by Prism aren't that meaningful anwyay.

3. I think that's not actually applicable anymore. See the reference below:

• Apparently Shiki used to be server-side only, that must be what you remember of it: https://github.com/shikijs/shiki/issues/91

How about Highlight.js?

I don't know, but at least it seems more under development than Prism is.

[Josh-Cena]

You mentioned docusaurus-preset-shiki-twoslash. It works by injecting a remark-shiki-twoslash plugin into our content plugin's config. This plugin is a Remark plugin, so it runs on server side. Basically, it takes a Markdown string and spits out a bunch of JSX. However, this transformation is done once and only once when the Markdown is compiled to JSX, but not if your code was authored in JSX to start with.

Now there could be a way to migrate all this logic to client side, sure; but that's not how any existing solution in the wild works, and it will need a lot of prototyping in the community before we can consider its feasibility to be recommended to everyone. There's nothing stopping you from implementing it yourself, or asking the Shiki folks to try it.

[felipecrs]

Alright. That makes sense!

Right now I will not be working on implementing it myself, and I believe Shiki folks are busy working towards Shiki 1.0. Maybe I can follow up with them after.

Thanks for the feedback, I'll close this issue for now (and reopen if something changes).

@SuperManito, I'd recommend you open a separate issue to discuss/propose Highlight.js.

[slorber]

There's also https://bright.codehike.org/

We'll figure out what is the best solution after migrating to server components.

For now the current solution is good enough and the system is flexible enough for you to use Shiki if you want.

[felipecrs]

There's also https://bright.codehike.org/

OMG that's amazing!

[ryanhamilton]

May I suggest you make a library agnostic way of overriding the code highlighter. For different languages people will always want different highlighters. If you give them an extensible method of simply plugging in their own, they will do it. My personal preference is codemirror :)

[slorber]

@ryanhamilton we have 4 ways of doing that:

• implementing your own theme
• providing a remark plugin that process code blocks in Markdown AST, possibly highlighting code at build time instead of runtime
• providing your own @theme/MDXComponents renderer for code blocks
• swizzling the built-in @theme/CodeBlock and modifying it to not use Prism for some languages

That's probably largely enough extension points, we are not coupled to Prism, it is only used in one theme, but you can swap it for something else at multiple levels

[felipecrs]

It looks like Shiki got a complete revamp for 1.0: https://github.com/shikijs/shiki/releases

Apparently things that were previously only server side can now run client side.

[SuperManito]

Shiki may become the industry's best solution for code highlighting.

[johnnyreilly]

I'm actually really keen on Shiki, but the Docusaurus integration of https://github.com/shikijs/twoslash/tree/main/packages/docusaurus-preset-shiki-twoslash unfortunately has issues that mean (much as I'd like to) I've been unable to make use of it on my own site at present:

https://github.com/shikijs/twoslash/issues/120

[felipecrs]

I don't think docusaurus-preset-shiki-twoslash was updated to work with Shiki 1.0 yet, nor Docusaurus 3.0.

[Josh-Cena]

I'm +1 on investigating integration, but I think it would greatly help us if there's a community proof-of-concept. I'm mainly interested because Shiki can do other fun things like showing type/linting errors.

[lachieh]

@Josh-Cena I currently have the slightly older shikiji-rehype plugin implemented on cosmonic.com/docs (example here with wit highlighting) and am in the process of adding the newer shiki-rehype for wasmcloud.com. It took a little wrangling, so if there's interest I can make it into a docusaurus preset with more easily configurable options.

[slorber]

It would be hard to make Shiki the default option of Docusaurus considering we probably want by default to be consistent between the regular code block and the live code block (both using Prism) and afaik live code block is not really an option (nor a good idea?) with Shiki.

---

We usually don't create official plugins for things we don't use ourselves.

However, it's probably worth it to explore how to use Shiki/Twoslash to enhance the experience on our own website, and create an opt-in plugin that users can turn on with some code block metastring on Docusaurus v3.x?

```ts shiki
const hello = "World"

// @module: esnext
// @filename: maths.ts
export function absolute(num: number) {
  if (num < 0) return num * -1;
  return num;
}
// @filename: hello.ts
const hello = "World"
//    ^?

If this opt-in syntax highlighter becomes successful, we could see how to add an option to make it the default in Docusaurus v4. 
(in v3, you could write a remark plugin that adds that meta string for you, although it would be less convenient it's still possible)

---

Note:
- Shiki recently released v1.0 (after merging back the Shikiji fork): https://github.com/shikijs/shiki/releases/tag/v1.0.0
- Twoslash also has a new repo/org/docs and is now integrated as a Shiki transformer: https://github.com/twoslashes/twoslash
- Both tools are in active development
- Shiki now provides a rehype plugin instead of a remark one like previously

---

@lachieh thanks for your proposal, I'm curious to see how you implement such preset. 

To be honest, I think Docusaurus is missing a few primitives to make it possible to implement this as a robust plugin or preset, so it's probably better if we work on it in the main repo, but I'd be curious to look at your solution.

[lachieh]

Thanks, @slorber. That all makes sense. I'll hold off on making the preset for now but if anyone finds their way here from a search, the PR below shows one way to integrate the @shiki/rehype plugin with Docusaurus 3.

https://github.com/wasmCloud/wasmcloud.com/pull/297/files

[userdocs]

@lachieh thanks, I got it working on my install as I wanted to use shiki over prism. Prism is just not as good as shiki when showing more complex snippets.

Here is an example of a 2k line bash script that would have multiple cosmetic issues with prism due to the bash parser getting confused as well as inconsistent highlighting but works fine with shiki.

https://userdocs.github.io/qbittorrent-nox-static/docs/bash/

Now, I understand this is not a normal use case posting a 2k lines of code but i tried many things/langs/examples with prism, I made my own themes and the outcome was never as good as it is with shiki. It's just better.

You can use any vscode json theme and it will look exactly like it does in vscode in docusaurus. I'm using this https://github.com/daltonmenezes/aura-theme/tree/main/packages/vscode

This is one reason I wanted to jump over to astro/starlight which is using this https://github.com/expressive-code/expressive-code + shiki but i'd really like to see it implemented here.

This vscode engine idea is doing something right. It looks amazing.

[slorber]

TIL about https://expressive-code.com, thanks that looks interesting. I'll investigate and try to figure out if we should use this instead of Shiki directly

[kubukoz]

TIL about expressive-code.com, thanks that looks interesting. I'll investigate and try to figure out if we should use this instead of Shiki directly

Hi, @slorber, any success with this?

[pomber]

Code Hike could be another alternative. There's an example of how to use it with Docusaurus (v3.5.0+) here: https://github.com/code-hike/examples/tree/main/with-docusaurus

[slorber]

Indeed, thanks for suggesting it

Also showing what can be built in userland! We don't necessarily need to build first-class support and the community can work on it too.

---

Note Josh Comeau provided useful feedback on Shiki:

https://www.joshwcomeau.com/blog/how-i-built-my-blog-v2/

As much as I love Shiki, it does have some tradeoffs.

Because it uses a more powerful syntax-highlighting engine, it’s not as fast as other options. I was originally rendering these blog posts “on demand”, using standard Server Side Rendering rather than static compile-time HTML generation, but found that Shiki was slowing things down quite a bit, especially on pages with multiple snippets. This problem can be solved either by switching to static generation or with HTTP caching.

Shiki is also memory-hungry; I ran into an issue with Node running out of memory(opens in new tab), and had to refactor to make sure I wasn't spawning multiple Shiki instances.

The biggest issue, however, is that sometimes I need syntax highlighting on the client. For example, in my Gradient Generator, the snippet changes based on how the user edits the shadows:There’s no way to generate this at compile-time, since the code is dynamic! For these cases, I have a second Shiki highlighter. This one is lighter, only supporting a small handful of languages.

It's possible that making Shiki the default for Docusaurus would slow down the build time and increase memory usage, in addition to the inability to use it in live code blocks. This confirms my initial intuition that we should keep Prism as the default highlighter for now, and Shiki should be an optional plugin.

[SuperManito]

Indeed, thanks for suggesting it

Also showing what can be built in userland! We don't necessarily need to build first-class support and the community can work on it too.

Note Josh Comeau provided useful feedback on Shiki:

https://www.joshwcomeau.com/blog/how-i-built-my-blog-v2/

As much as I love Shiki, it does have some tradeoffs.

Because it uses a more powerful syntax-highlighting engine, it’s not as fast as other options. I was originally rendering these blog posts “on demand”, using standard Server Side Rendering rather than static compile-time HTML generation, but found that Shiki was slowing things down quite a bit, especially on pages with multiple snippets. This problem can be solved either by switching to static generation or with HTTP caching.

Shiki is also memory-hungry; I ran into an issue with Node running out of memory(opens in new tab), and had to refactor to make sure I wasn't spawning multiple Shiki instances.

The biggest issue, however, is that sometimes I need syntax highlighting on the client. For example, in my Gradient Generator, the snippet changes based on how the user edits the shadows:There’s no way to generate this at compile-time, since the code is dynamic! For these cases, I have a second Shiki highlighter. This one is lighter, only supporting a small handful of languages.

It's possible that making Shiki the default for Docusaurus would slow down the build time and increase memory usage, in addition to the inability to use it in live code blocks. This confirms my initial intuition that we should keep Prism as the default highlighter for now, and Shiki should be an optional plugin.

Shiki can sync use now #sync-usage

By the way. Shiki can't use on Docusaurus v3

[lachieh]

@SuperManito There are plenty of examples of Shiki on Docusaurus 3, including in this thread. Are you having a particular issues with it?

[SuperManito]

@SuperManito There are plenty of examples of Shiki on Docusaurus 3, including in this thread. Are you having a particular issues with it?

The plugin docusaurus-preset-shiki-twoslash doesn't support Docusaurus 3.

[lachieh]

That's correct. It's a community preset that was created with the pre-1.0 version of Shiki.js before maintainership was taken over by Anthony Fu. There is already an issue about that on the old repo. That change is unlikely to happen since Shiki 1.0 is already supported as a rehype plugin, and therefore can be implemented directly into Docusaurus 3.0.

[Sainan]

I have yet to see any examples of Shiki on Docusaurus 3 that actually just work.

[lachieh]

Here is a guide on installing @shiki/rehype with the latest version of docusaurus. The only necessary part is the first section of which the TL;DR is "install the packages and add the plugin to your config".

https://lachieh.github.io/docusaurus-with-shiki-rehype/docs/intro

The following section shows how to add a dark/light mode, and the site itself includes examples on how to use some of the transformers from @shiki/common-transformers.

[slorber]

Thanks for the demo @lachieh

It's a good starting point so that we can think about how we could simplify this setup in the future

[lachieh]

@slorber yeah, no worries. Is this something that could be bundled into a theme or a preset? Which would work better assuming most would likely try to install alongside preset-classic?

[slorber]

We plan to introduce a global markdown config and plugin customization API to register global remark/rehype plugins, so this can be implemented as a plugin. We can figure out a solution where our theme components offer a "pass-through" mode so that you don't have to swizzle them to unplug Prism runtime code blocks.

[lachieh]

I'd be interested in contributing to that effort!

[Nova38]

I am running into a import error about some of the submodules not exporting the main. When I tried your example in stackblitz it failed to run. Running it locally with yarn it worked. I saw that you had to patch one of the libraries. Any idea why this error is happening and if it lies with the way docusaurus is importing the library or if it is an issue with they way shiki is exporting itself. I am trying to use this in a project that is using npm so it isn't possible to directly patch the library with our using an external package, but manually patching it in the node_modules folder it had the same issue in another one of the sub libraries

Here is a guide on installing @shiki/rehype with the latest version of docusaurus. The only necessary part is the first section of which the TL;DR is "install the packages and add the plugin to your config".

https://lachieh.github.io/docusaurus-with-shiki-rehype/docs/intro

The following section shows how to add a dark/light mode, and the site itself includes examples on how to use some of the transformers from @shiki/common-transformers.

[felipecrs]

I am running into a import error about some of the submodules not exporting the main.

This one?

[ERROR] Error: Docusaurus could not load module at path "/home/felipecrs/repos/website/docusaurus.config.ts"
Cause: No "exports" main defined in /home/felipecrs/repos/website/node_modules/@shikijs/vscode-textmate/package.json
    at loadFreshModule (/home/felipecrs/repos/website/node_modules/@docusaurus/utils/lib/moduleUtils.js:36:15)
    at loadSiteConfig (/home/felipecrs/repos/website/node_modules/@docusaurus/core/lib/server/config.js:36:62)
    ... 6 lines matching cause stack trace ...
    at async file:///home/felipecrs/repos/website/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 {
  [cause]: Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /home/felipecrs/repos/website/node_modules/@shikijs/vscode-textmate/package.json

Because this is what I get when I try it. I believe it has something to do with my package.json having type: module. @lachieh, any ideas?

[lachieh]

I thought i had removed that patch actually, sorry about that.

Did some digging, it should be a pretty simple solve. Docusaurus uses jiti to enable esm/cjs/json/ts imports but isn't able to import @shikijs/vscode-textmate for some reason. This was resolved in jiti@v2 so I'll open a PR to update that dependency.

Nov 25 Update: Created a PR which is waiting on a potential bug in jiti.