Allow Highlighter to be loaded synchronously when theme and langs are injected instead of dynamically loaded

[fuxingloh]

I'm not sure if this is being worked on for v1, but I wrote it here so we could possibly address it for v1.

Currently, there are "two issues" preventing shiki from being used indiscriminately on NextJS (or any other modern framework that uses webpack) on the client, build, or server (CSR, SSR, SSG, ISR) as pointed out in https://github.com/shikijs/shiki/issues/138#issuecomment-1057471160. The "plug and play" way without any modification is forcing the Highlighter to be run on "build-time, on server" (aka SSG as seen in this example https://github.com/shikijs/next-shiki/blob/main/pages/index.tsx), but this heavily restricts ISR and dynamic content.

The "two issues":

1. async components

is not compatible (as of React 18.2) on client; through the use of suspense and useEffect, we could get around it and create a component such as <ShikiHighlighter lang="html"> but this hurts SEO since the initial state would be empty. If a react component would be designed, it should cater to run indiscriminately on build, server, and client.

export function ShikiHighlighter(props: {  children: string;  lang: string }) {
-  const highlighter = await getHighlighter();
+  const highlighter = getHighlighter();
  const tokens = highlighter.codeToThemedTokens(props.children, props.lang);
  const html = renderToHtml(tokens, {
    elements: {
      pre({ className, style, children }) {
        return `<pre class="${className} css-variable" style="${style}">${children}</pre>`;
      },
    },
  });

  return <div className={props.className} dangerouslySetInnerHTML={{ __html: html }} />;
}

Essentially, DX would be optimal if we have a method called getHighlighterSync. However, this opens up another issue: _fetchAssets.

2. untraceable _fetchAssets

to deterministically bundle required assets with webpack or included with https://github.com/vercel/nft. This prevents the necessary theme and lang from being loaded due to missing files via fs.promise.readFile or unbundled import bundled. We cannot load the Highlighter without customizing the loader architect (setCDN, paths: {}) to load seamlessly on the server or client.

By injecting the known Grammar and Theme for my PR https://github.com/levaintech/frontmatter/pull/39, I could get around the bundling issue:

import { getHighlighter, Highlighter, Lang, renderToHtml } from 'shiki';
import JsonGrammar from 'shiki/languages/json.tmLanguage.json';
import CssVariablesTheme from 'shiki/themes/css-variables.json';

let Highlighter: Highlighter;

export async function highlight(props: { code: string; language: Lang }): Promise<string> {
  if (highlighter === undefined) {
    highlighter = await getHighlighter({
      theme: CssVariablesTheme as any,
      langs: [
        {
          id: 'json',
          scopeName: 'source.json',
          path: '-',
          grammar: JsonGrammar as any,
        },
      ],
    });
  }

  const tokens = highlighter.codeToThemedTokens(props.code, props.language);
  return renderToHtml(tokens);
}

Some ideas: Maybe we could convert _fetchAssets to a non-promise and allow the assets to be bundled or traceable by

• including 1 default theme and top 10 languages for the default bundled Highlighter (using require() or import instead of node:fs). This would result in at least 1 MB of assets with a default bundled shiki
• or upstream to pick the theme and language they will need and import them import { CssVariable } from 'shiki/themes' and tree shaking away the rest in production

[muuvmuuv]

Hey, I think this would also improve esbuild/Vite bundling since dynamic imports aren't working in pre-bundling here. See also Iconicons issue: https://github.com/ionic-team/ionicons/issues/1032 which now requires addIcons to manually inject them

[antfu]

Not only the fetch but also the grammar parsing are required to be async.

For 2. and the bundling, you can try https://github.com/antfu/shikiji (which is proposed in https://github.com/shikijs/shiki/issues/510)

[antfu]

It's not possible for Shiki to runs in sync context. The getHightlighter is a solution to separate the async and sync parts.

For 2. you can try the new v1.0 now, #557

[thesoftwarephilosopher]

It's not possible for Shiki to runs in sync context.

Why can't you separate out the potentially-async logic to load data from the definitely-sync logic to process it?

I have a project that's entirely sync, and I can't use shiki for it, not even 0.14.3. What a shame, it highlights tsx so well.