creating a shiki wrapper for nextjs, and rsc

saadi925 commented 1 month ago

Clear and concise description of the problem

The Shiki is Vue and Nuxt first , it also works fine with the next js , but there are a few problems with it when using in nextjs. it just does not support it the way it works with vue. and we have too import different functions and each have their own set of rules.

Suggested solution

Therefore , I am working on a shiki nextjs, react , rsc wrapper , i am following the best practices, using type guards, and i want to know what suggestions community will give. i will also write the docs (in nextjs with turborepo,fuma), i am following a pattern of single config. you only have to create a function generateShiki. actions have the actions like 'codeToHtml' | 'codeToHast' | 'codeToTokens' | 'codeToTokensBase' | 'codeToTokensWithThemes'; etc there are some base options , in each action we can overide the default options , and each action have it's appropriate options. to use a theme in the base options we have to load it in the custom highlight options, actions is an array on action , if we remove an action and invoke their function will be undefined.

import generateShiki, { RenderCode } from '@repo/ui/shiki/react'
import oneDark from 'shiki/themes/one-dark-pro.mjs';
import oneLight from 'shiki/themes/one-light.mjs';
export default async function Page() {
  const result = await generateShiki({
    code: ` const x = 10
console.log(x)`
    , baseOptions: {
      lang: 'javascript',
      theme: "one-dark-pro",
    },
    customHighlighterOptions: {
      themes: [oneDark, oneLight],
    },

    actions: [
      {
        action: 'codeToHtml',
        options : {
          decorations : []
        }
      },
      {
        action: 'codeToTokensWithThemes',
        options: {
          themes: {
            light: "one-light",
            dark: "one-dark-pro"
          }
        }
      }

    ],
  });
  const html = await result.renderCode?.();
  const hast = await result.getHast?.();
  const tokensBase = await result.getTokensBase?.();
  const tokensFull = await result.getTokens?.();
  const tokens_theme = await result.getTokensWithThemes?.();
// the function will be undefined , whose action is not listed in the actions array , using only what it needs 
  return (
    <main>
      <RenderCode className="" code={html} />

    </main>
  );
}

we do not have to write that much to get started , it is an example so i did it intentionally to explain the structure. we can start with just writting

import generateShiki, { disposeHighlighter, RenderCode } from '@repo/ui/shiki/react'
export default async function Page() {
  const result = await generateShiki({
    code: `console.log('Hello, world!')`
    , baseOptions: {
      lang: 'javascript',
      theme: "nord",
    },
    actions: [
      {
        action: 'codeToHtml',
      },
    ],
  });
  const html = result.renderCode?.();
  disposeHighlighter()
  return (
    <main>
      <RenderCode code={html || ""} />
    </main>
  );
}

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Contributes

[x] If this feature request is accepted, I am willing to submit a PR to fix this issue

repraze commented 1 month ago

I've been experimenting with shiki and next.js with app router for the last few days. So far I have been unable to get SSR working for dynamic pages. Only static build or client side highlighting works.

When creating a highlighter at runtime, it fails to dynamically import the languages. (From other issues, it seems next.js might not support the kind of import shiki is doing) I've tried to bypass that by using the core highlighter and playing with the imports, but no luck so far.

Would this wrapper/enhancement help with that use case? Otherwise next.js seems to work for me on client side / static site generation.

saadi925 commented 1 month ago

yeah

I've been experimenting with shiki and next.js with app router for the last few days. So far I have been unable to get SSR working for dynamic pages. Only static build or client side highlighting works.

When creating a highlighter at runtime, it fails to dynamically import the languages. (From other issues, it seems next.js might not support the kind of import shiki is doing) I've tried to bypass that by using the core highlighter and playing with the imports, but no luck so far.

Would this wrapper/enhancement help with that use case? Otherwise next.js seems to work for me on client side / static site generation.

yeah i have been working on it , it could work even now with ssr for dynamic pages .

i have created this wrapper , but i am still working on it , this will be an optimized single function . which could be used for csr, static site and ssr ,

/**
 * Base options for code highlighting
 */
export interface BaseOptions {
  lang: sh.BundledLanguage;
  theme: sh.BundledTheme;
}

/**
 * Action options for different code highlighting methods
 */
// Union of all action types
export type ShikiAction =
  | CodeToHtmlAction
  | CodeToHastAction
  | CodeToTokensAction
  | CodeToTokensBaseAction
  | CodeToTokensWithThemesAction;

/**
 * Props for the generateShiki function
 */
export interface ShikiConfig {
  baseOptions: BaseOptions;
  actions: ShikiAction[];
  customHighlighterOptions?: HighlighterCoreOptions;
}

/**
 * Result type for the generateShiki function
 */

// Result Types
export type ShikiResult = {
  codeToHtml?: (code: string, options?: CodeToHtmlAction["options"]) => string
  getTokensBase?: (code: string, options?: CodeToTokensBaseAction["options"]) => Promise<sh.ThemedToken[][]>
  getTokens?: (code: string, options?: CodeToTokensAction["options"]) => Promise<sh.TokensResult>
  getHast?: (code: string, options?: CodeToHastAction["options"]) => Promise<any>
  getTokensWithThemes?: (code: string, options?: CodeToTokensWithThemesAction["options"]) => Promise<sh.ThemedTokenWithVariants[][]>;
}

Example

i am also working on dynamic languages import without loosing optimizations this is a server component the only loaded language is javascript and we are loading tsx language dynamically without loading the full bundle . it's just loading that particular language

import generateShiki, { disposeHighlighter, RenderCode, ShikiConfig } from '@repo/ui/shiki/react'
import minDark from "shiki/themes/min-dark.mjs"
import oneDark from "shiki/themes/one-dark-pro.mjs"
export default async function Page() {

const config : ShikiConfig  ={
  baseOptions: {
    lang: 'javascript',
    theme:"min-dark"
  },
  customHighlighterOptions  : {
    themes : [minDark, oneDark]
  },
  actions : [
    {
      "action": "codeToHtml",
    }
  ]
}
  const hl = await generateShiki(config);
// example code
 const code1 =   `
import React from 'react'
import { ThemeProvider } from 'next-themes'
`
const c =  await hl.codeToHtml?.(code1, {
  "lang" :"tsx",
})

if a language is not found it moves back to the default one , instead of error
  disposeHighlighter()
  return (
    <main >
      <div 
      dangerouslySetInnerHTML={{
        __html : c || ""
      }}
      className={`
      py-2 mx-12 mt-2 px-4 
      rounded-md overflow-x-auto 
      bg-[#1f1f1f]
      `} />
    </main>
  );
}

also see this article must read last lines , final thoughts shiki_nextjs

repraze commented 1 month ago

Sounds promising, but could be hard to maintain the mapping on top of shiki.

I found that importing from @shikijs/core helps with next js runtime since shiki bundling is the issue on the server. The only thing left on my side would be loading the langs/themes around dynamic imports.

import {createHighlighterCore} from "@shikijs/core";
import getWasm from "@shikijs/core/wasm-inlined";

async function getHighlighter (){
    const highlighter = await createHighlighterCore({loadWasm: getWasm});

    highlighter.loadLanguage(/* can't use import("shiki/langs/javascript.mjs") */);

    return highlighter;
}

saadi925 commented 1 month ago

Sounds promising, but could be hard to maintain the mapping on top of shiki.

I found that importing from @shikijs/core helps with next js runtime since shiki bundling is the issue on the server. The only thing left on my side would be loading the langs/themes around dynamic imports.
import {createHighlighterCore} from "@shikijs/core";
import getWasm from "@shikijs/core/wasm-inlined";

async function getHighlighter (){
    const highlighter = await createHighlighterCore({loadWasm: getWasm});

    highlighter.loadLanguage(/* can't use import("shiki/langs/javascript.mjs") */);

    return highlighter;
}

i am not doing any mapping on it , the mappings are already in the shiki package , the are using it it is very great, they have the mappings in the langs of all the imports i am importing it dynamically on server side (demand) . the above article link states that it is not possible and easy but it is . i am using it , it's working fine.

fuma-nama commented 3 weeks ago

Cannot quite understand the need of this feature, it's working great on the latest Next.js 14 release. I believe Next.js has fixed the issues with Shiki even if there was an issue on older versions.

Note that Next.js added shiki to serverExternalPackages by default

asmyshlyaev177 commented 3 weeks ago

That approach works locally with NextJS14 Server components, but not on Vercel.

import {
  transformerNotationHighlight,
  transformerNotationWordHighlight,
} from '@shikijs/transformers';
import { type HighlighterCore } from 'shiki';
import { createHighlighterCore } from 'shiki/core';
import tsLang from 'shiki/langs/typescript.mjs';
import githubTheme from 'shiki/themes/github-dark.mjs';
import getWasm from 'shiki/wasm';

export const createHighlighter = async () =>
  await createHighlighterCore({
    themes: [githubTheme],
    langs: [tsLang],
    loadWasm: getWasm,
  });

export let highlighter: HighlighterCore

export const highlight = async (content: string) => {
  if (!highlighter) {
    highlighter = await createHighlighter()
  }
  return highlighter?.codeToHtml?.(content, {
    lang: 'typescript',
    theme: 'github-dark',
    transformers: [
      transformerNotationHighlight(),
      transformerNotationWordHighlight(),
    ],
  }) || '';
}

and use like this

import { highlight } from '../highlighter';

export const File = async ({
  content,
}: {
  content: string;
}) => {

  const text = await highlight(content);

  return (
      <code>
        <pre
          dangerouslySetInnerHTML={{ __html: text }}
          className=" dark:bg-gray-800 p-4"
        ></pre>
      </code>
  );
};

@fuma-nama need to use "Fine-grained bundle", with edge function bundle size is very limited, from docs couldn't find a proper way to do it with Next.js.

fuma-nama commented 3 weeks ago

Production build works as expected: https://shiki-test-eight.vercel.app It uses revalidate = 0 which forced the page into dynamic rendered.

edge function bundle size

You're not supposed to use Shiki in edge runtime, it uses Node.js APIs. And honestly, you have no reasons to use Shiki under edge environments like Middleware, for normal pages, serverless is absolutely enough

asmyshlyaev177 commented 3 weeks ago

Production build works as expected: https://shiki-test-eight.vercel.app It uses revalidate = 0 which forced the page into dynamic rendered.

edge function bundle size

You're not supposed to use Shiki in edge runtime, it uses Node.js APIs. And honestly, you have no reasons to use Shiki under edge environments like Middleware, for normal pages, serverless is absolutely enough

Ok, better to add more exhaustive docs for Next.js, still source of confusion.

fuma-nama commented 3 weeks ago

Hmm I see, maybe open a separate issue? Would be nice to have a Next.js guide under the integrations section.

At least from what I see, the original issue of OP has been solved on newer versions of Next.js.

shikijs / shiki