1.18.0 seems causing a breaking change for highlight in some case

arvinxx commented 3 days ago

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Describe the bug

Thank you for bring this amazing work! LobeChat have this beautiful code highlight all credit to you~

But it seems there is a breaking change for 1.18.0 which case the code highlight invalid.

So I pin to 1.17.7 temporarily with the PR to fix it.

Our usage code is:

import { useThemeMode } from 'antd-style';
import { Loader2 } from 'lucide-react';
import { memo } from 'react';
import { Flexbox } from 'react-layout-kit';

import Icon from '@/Icon';
import { useHighlight } from '@/hooks/useHighlight';
import { DivProps } from '@/types';

import { useStyles } from './style';

export interface SyntaxHighlighterProps extends DivProps {
  children: string;
  language: string;
}

const SyntaxHighlighter = memo<SyntaxHighlighterProps>(
  ({ children, language, className, style }) => {
    const { styles, cx } = useStyles();
    const { isDarkMode } = useThemeMode();

    const { data, isLoading } = useHighlight(children.trim(), language, isDarkMode);

    return (
      <>
        {isLoading || !data ? (
          <div className={cx(styles.unshiki, className)} style={style}>
            <pre>
              <code>{children.trim()}</code>
            </pre>
          </div>
        ) : (
          <div
            className={cx(styles.shiki, className)}
            dangerouslySetInnerHTML={{
              __html: data as string,
            }}
            style={style}
          />
        )}
        {isLoading && (
          <Flexbox
            align={'center'}
            className={styles.loading}
            gap={8}
            horizontal
            justify={'center'}
          >
            <Icon icon={Loader2} spin />
          </Flexbox>
        )}
      </>
    );
  },
);

export default SyntaxHighlighter;

and the code in useHighlight is:

import {
  transformerNotationDiff,
  transformerNotationErrorLevel,
  transformerNotationFocus,
  transformerNotationHighlight,
  transformerNotationWordHighlight,
} from '@shikijs/transformers';
import { type Highlighter, getHighlighter } from 'shiki';
import useSWR from 'swr';

import { themeConfig } from '@/Highlighter/theme';

import languageMap from './languageMap';

export const FALLBACK_LANG = 'txt';

const FALLBACK_LANGS = [FALLBACK_LANG];

let cacheHighlighter: Highlighter;

const initHighlighter = async (lang: string): Promise<Highlighter> => {
  let highlighter = cacheHighlighter;
  const language = lang.toLowerCase();

  if (highlighter && FALLBACK_LANGS.includes(language)) return highlighter;

  if (languageMap.includes(language as any) && !FALLBACK_LANGS.includes(language)) {
    FALLBACK_LANGS.push(language);
  }

  highlighter = await getHighlighter({
    langs: FALLBACK_LANGS,
    themes: [themeConfig(true), themeConfig(false)],
  });

  cacheHighlighter = highlighter;

  return highlighter;
};

export const useHighlight = (text: string, lang: string, isDarkMode: boolean) =>
  useSWR(
    [lang.toLowerCase(), isDarkMode ? 'dark' : 'light', text].join('-'),
    async () => {
      try {
        const language = lang.toLowerCase();
        const highlighter = await initHighlighter(language);
        const html = highlighter?.codeToHtml(text, {
          lang: languageMap.includes(language as any) ? language : FALLBACK_LANG,
          theme: isDarkMode ? 'dark' : 'light',
          transformers: [
            transformerNotationDiff(),
            transformerNotationHighlight(),
            transformerNotationWordHighlight(),
            transformerNotationFocus(),
            transformerNotationErrorLevel(),
          ],
        });
        return html;
      } catch {
        return '';
      }
    },
    { revalidateOnFocus: false },
  );

export { default as languageMap } from './languageMap';

the source is here: https://github.com/lobehub/lobe-ui/blob/master/src/hooks/useHighlight.ts#L41

I dig it a little and find that the dom seems different:

v1.17.7 's dom:

but in the v1.18.0's dom:

it means that in the v1.18.0 the control flow is go to the condition of `isLoading || !data ? ( <div className={cx(styles.unshiki, className)} style={style}>

              {children.trim()}

      </div>
    )`.

and then I try to log the error, and it show up with:

ShikiError: Language `ts` not found, you may need to load it first
    at Object.getLanguage (mf-dep____vendor.6f98842d.async.js:57165:13)
    at codeToTokensBase (mf-dep____vendor.6f98842d.async.js:56270:29)
    at codeToTokens (mf-dep____vendor.6f98842d.async.js:56559:14)
    at codeToHast (mf-dep____vendor.6f98842d.async.js:56622:7)
    at codeToHtml (mf-dep____vendor.6f98842d.async.js:56842:74)
    at Object.codeToHtml (mf-dep____vendor.6f98842d.async.js:57271:37)
    at _callee2$ (useHighlight.ts:48:1)
    at tryCatch (mf-dep____vendor.6f98842d.async.js:35914:16)
    at Generator.<anonymous> (mf-dep____vendor.6f98842d.async.js:36002:17)
    at Generator.next (mf-dep____vendor.6f98842d.async.js:35943:21)
    at asyncGeneratorStep (mf-dep____vendor.6f98842d.async.js:35533:24)
    at _next (mf-dep____vendor.6f98842d.async.js:35552:9)
lockdown-install.js:1

Why it's worked in v1.17.7 but breaking in the 1.18.0 ?

Reproduction

https://github.com/lobehub/lobe-chat/tree/reproduction/shiki-1.18.0

Contributes

[ ] I am willing to submit a PR to fix this issue
[ ] I am willing to submit a PR with failing tests

fuma-nama commented 6 hours ago

Hmm I'm not quite sure what does the logic in initHighlighter do. Normally to lazy load required languages, we use async highlighter.loadLanguage function in advanced of codeToHast/codeToHtml.

Here's my example tested in CodeSandbox:

"use client";

import { useEffect, useState } from "react";
import { BundledLanguage, createHighlighter } from "shiki";

const snippets = [
  {
    lang: "ts",
    code: `export const hello: string = "Hello"`,
  },
  {
    lang: "java",
    code: `System.out.println("Hello")`,
  },
  {
    lang: "py",
    code: `print(f"Hello")`,
  },
];

export default function Page() {
  return <div>{snippets.map(item => <Highlight key={item.lang} {...item} />)}</div>;
}

const highlighter = createHighlighter({
  langs: ["ts"], // preload language
  themes: ["vitesse-dark"],
});

function Highlight({ code, lang }: { code: string; lang: string }) {
  const [html, setHtml] = useState<string>();

  useEffect(() => {
    // TEST: do it on browser only
    async function run() {
      const instance = await highlighter;
      if (!instance.getLoadedLanguages().includes(lang)) {
        console.log('load', lang)
        // the language may not exist, we assume it is always valid
        await instance.loadLanguage(lang as BundledLanguage);
      }

      setHtml(instance.codeToHtml(code, {
        lang,
        theme: 'vitesse-dark'
      }))
    }
    void run()
  }, [code, lang]);

  if (!html) return null
  return <div dangerouslySetInnerHTML={{ __html: html }} />
}

Notice that the function codeToHast/codeToHtml exported from shiki does lazy load by default, this is only required for highlighter API.

As you are using React, I would suggest you to check https://shiki.style/packages/next#custom-components, which illustrates the way to render JSX instead of HTML with Shiki.

antfu commented 40 minutes ago

Shiki v1.18 does not introduce any functionality changes (https://github.com/shikijs/shiki/compare/v1.17.7...v1.18.0). The only thing notable is https://github.com/shikijs/shiki/pull/781 - which seems not related to your case.

The example and repo has too much React related logic for me to take a look, can you provide a minimal reproduction instead?

shikijs / shiki