search

shikijs / shiki

A beautiful yet powerful syntax highlighter
http://shiki.style/
MIT License
10.26k stars 375 forks source link

Usage with Next.js SSR #138

Closed schickling closed 3 years ago

schickling commented 3 years ago

First of all: Thanks so much for creating this great library. I've spend too many hours trying to configured Monaco to get VSC themes working before I found Shiki. 😅

I'm trying to use Shiki in a Next.js app using Next.js' SSR feature via getStaticProps. Everything works great during local development (via next dev) however the app breaks in production after running next build (which bundled the app) resulting in errors like these:

image

The reason for this is that Shiki dynamically tries to access the theme/grammar files using the fs Node package. However these resources are no longer available after the next build step. Next.js uses nft to automatically detect and bundle assets. It would be great if Shiki could access the required assets in a way that nft can understand it and it therefore works with Next.js.

As a temporary workaround I'm doing the following:

const theme = require('shiki/themes/github-light.json')
const grammar = require('shiki/languages/tsx.tmLanguage.json')

const highlighter = await getHighlighter({
  theme,
  langs: [{ id: 'tsx', scopeName: 'source.tsx', grammar }],
})
odex21 commented 3 years ago

I had a similar problem when I used with vitesse. Your workaround is worked for me.

octref commented 3 years ago

I'm not very familiar with Next.js, but I gave it a try. next build and next start seems to work fine, can you tell me what's different in your setup?

https://github.com/octref/next-shiki/commit/dd83217ad9f276c8d4849882b7b3f174d77a47b9

kennetpostigo commented 3 years ago

@octref do you know if there is a way to get mdx in nextjs to pick use shiki highlighting?

octref commented 3 years ago

@kennetpostigo You might need to write a remark plugin: https://nextjs-prism.vercel.app/prism

THiragi commented 3 years ago

I had the same problem with my Next.js app (and dealt with it in the same way).

Everything works fine in local using next build and next start. I believe that bundled the app access the theme/language files in local node_modules. So it doesn't work in the production environment without node_modules (e.g. Vercel).

Even in local production server started with next start, if you do npm uninstall shiki after next build, rendering with Shiki will fail. (This means that theme/language files are not included in bundled the app.)

octref commented 3 years ago

@THiragi The currently recommended way to include language/theme JSON files are not bundling them, but serving them as static assets.

octref commented 3 years ago

See https://github.com/octref/next-shiki/commit/dd83217ad9f276c8d4849882b7b3f174d77a47b9 for next.js usage. If you still have issues please open a new one.

thien-do commented 2 years ago

See https://github.com/octref/next-shiki/commit/dd83217ad9f276c8d4849882b7b3f174d77a47b9 for next.js usage. If you still have issues please open a new one.

@octref the next-shiki repo will always work, because the Shiki highlighting is done at getStaticProps, which is run at build-time, which has the full node_modules there.

However, I think it's not what @schickling and others are asking for in this issue. They are asking for SSR, which is run-time, and on server-side.

(basically in next-js there are 3 states:

That being said, I think I successfully have Shiki on getServerSideProps! This means:

The trick is:

The key here is to let Vercel nft to know about the existence of Shiki/themes and shiki/languages so they are included in the production run-time

Sample code: https://github.com/thien-do/memos.pub/blob/a3babb1f149f05c43012278331f885d81f5fcfac/lib/mdx/plugins/code.ts

ng-hai commented 2 years ago

It’s great if Shiki allows to fetch languages and themes from CDNs like unpkg.com/shiki even in server side.

rfoel commented 2 years ago

I'm using remix and had the same problem. I'm using it with serverless with esbuild and I had to mark shiki as an external module to bundle it with my application. Also, this comment explains well what's going on.

sreetamdas commented 1 year ago

For anyone looking for a run-time, client-side solution to use Shiki in Next.js, this is now possible with setWasm and setCDN.

To be clear, this allows running Shiki on the client-side, which will add ~137kB for onig.wasm and an additional ~15-20kB per language grammar. You most likely want to use it on the server during run-time instead: and for this @thien-do's solution above works well enough, or you can check out this official example: https://github.com/shikijs/next-shiki

Anyway, here's what you can do: (I used https://github.com/shikijs/shiki-playground as my main point of reference)

const preloadedLangs: Array = ["js", "jsx", "ts", "tsx", "elixir"]; export async function getClientSideHighlighter() {

And that's mostly it: you can now use Shiki to get your tokenized HTML and put it inside a dangerouslySetInnerHTML.

alivault commented 1 year ago

See octref/next-shiki@dd83217 for next.js usage. If you still have issues please open a new one.

@octref the next-shiki repo will always work, because the Shiki highlighting is done at getStaticProps, which is run at build-time, which has the full node_modules there.

However, I think it's not what @schickling and others are asking for in this issue. They are asking for SSR, which is run-time, and on server-side.

(basically in next-js there are 3 states:

  • build-time, on server -> this is getStaticProps, your example
  • run-time, on server -> this is getServerSideProps, or getStaticProps in case of ISR, which is this issue
  • run-time, on client -> not those getXXX at all)

That being said, I think I successfully have Shiki on getServerSideProps! This means:

  • ALL languages are available
  • ALL themes are available
  • NONE of them are bundled into client side

The trick is:

  • Copy shiki/themes and shiki/languages to somewhere outside of node_modules, maybe under lib/shiki
  • Touch these folders in a server side function (e.g. fs.readdirSync)
  • Done!

The key here is to let Vercel nft to know about the existence of Shiki/themes and shiki/languages so they are included in the production run-time

Sample code: https://github.com/thien-do/memos.pub/blob/a3babb1f149f05c43012278331f885d81f5fcfac/lib/mdx/plugins/code.ts

This is the answer. Thank you so much. I finally got Rehype Pretty Code / Shiki working on Vercel thanks to your answer.

JoshMerlino commented 1 year ago

Im running NextJS 13.5 and using the app router. Nothing I tried here was working, Vercel refused to see the shiki languages regardless of what I did. After slamming my head for a week, I realized Vercel can download the stuff it needs during runtime in order to work with SSR.

This is a modified getHighlighter function which will download the theme and languages to the /tmp dir on Vercel.

import { existsSync } from "fs";
import { writeFile } from "fs/promises";
import { mkdirp } from "mkdirp";
import { tmpdir } from "os";
import { dirname, join } from "path";
import type { HighlighterOptions, ILanguageRegistration, Theme } from "shiki";
import { BUNDLED_LANGUAGES, getHighlighter as shiki_getHighlighter } from "shiki";

export async function getHighlighter({ theme, langs: _langs = BUNDLED_LANGUAGES.map(a => a.id), ...options }: Omit<HighlighterOptions, "langs" | "themes"> & { langs?: Array<ILanguageRegistration | string>, theme: Theme }) {

    // Ensure the tmp directory exists
    const TMP = process.env.VERCEL ? "/tmp" : tmpdir();

    // Convert all strings to languages
    const langs = _langs.map(lang => typeof lang === "string" ? BUNDLED_LANGUAGES.find(({ id, aliases }) => aliases?.map(a => a.toLowerCase()).includes(lang.toLowerCase()) || id === lang) : lang) as ILanguageRegistration[];

    // Get all dependencies
    const deps = langs.flatMap(lang => (lang.embeddedLangs || []).map(a => BUNDLED_LANGUAGES.find(b => b.id === a)));

    return await shiki_getHighlighter({

        ...options,

        paths: { languages: TMP },

        // Download the theme
        theme: await fetch(`https://unpkg.com/shiki/themes/${ theme }.json`, { redirect: "follow" })
            .then(res => res.json()),

        // Download all languages
        langs: await Promise.allSettled(Array.from(new Set([ ...langs, ...deps ])).map(async function(lang) {
            if (!lang) return;

            // Get output path
            const path = join(TMP, `${ lang.id }.tmLanguage.json`);
            if (!existsSync(dirname(path))) await mkdirp(dirname(path));

            // If the file exists, return it
            if (existsSync(path)) return { ...lang, path };

            await fetch(`https://unpkg.com/shiki/languages/${ lang.id }.tmLanguage.json`, { redirect: "follow" })
                .then(res => res.text())
                .then(grammar => writeFile(path, grammar, "utf-8"));

            return { ...lang, path };

        }))
            .then(langs => langs.filter(({ status }) => status === "fulfilled"))
            .then(langs => (langs as { value: ILanguageRegistration }[]).map(({ value }) => value)),

    });

}

Then with NextJS App Router, I can use

export default async function Page() {
    const code = `\
import Link from "next/link";
import { Card } from "nextui/Card";
import ogs from "open-graph-scraper";

export async function OpenGraphLink({ href }: { href: string; }) {
    const { result, error } = await ogs({ url: href });
    if (error) return null;

    const url = new URL(result.requestUrl || result.ogUrl || href);
    const name = result.twitterTitle || result.ogSiteName || result.ogTitle || url.hostname;
    const banner = result.ogImage?.[0]?.url || result.twitterImage?.[0]?.url;
    const description = result.ogDescription || result.twitterDescription;
    const favicon = (result.favicon?.startsWith("/") ? \`\${ url.protocol }//\${ url.hostname }/\${ result.favicon }\` : result.favicon) || \`\${ url.protocol }//\${ url.hostname }/favicon.ico\`;

    return (
        <Link href={ url.toString() } target="_blank">
            <Card className="p-0 gap-0 hover:shadow-lg dark:hover:shadow-xl hover:bg-gray-50 dark:hover:bg-gray-700/50 transition-all">
                {banner && <img alt={ name } className="object-cover m-[1px] rounded-md" src={ banner } />}
                <div className="flex gap-4 items-center mx-4 my-2">
                    <div className="rounded-full bg-gray-100 flex items-center justify-center w-10 aspect-square shrink-0">
                        <img alt={ name } height={ 32 } src={ favicon } width={ 32 } />
                    </div>
                    <div className="flex flex-col relative overflow-hidden max-w-full">
                        <h1 className="text-lg font-medium text-gray-800 dark:text-gray-200 truncate -mb-0.5">{name}</h1>
                        <p className="text-sm whitespace-normal">{\`\${ url.hostname }\${ url.port }\${ url.pathname }\`.replace(/\\/$/, "")}</p>
                    </div>
                </div>
                <p className="!p-0 !px-4 !pb-2.5 text-sm text-gray-800 dark:text-gray-200">{description}</p>
            </Card>
        </Link>
    );
}`;

    const highlighter = await getHighlighter({ theme: "github-dark", langs: [ "tsx" ]});
    const __html = highlighter.codeToHtml(code, "tsx");

    return <div className="[&>pre]:whitespace-pre-wrap" dangerouslySetInnerHTML={{ __html }} />;

}

Want to use it with MDX?

rehypePlugins: [
        [ rehypePrettyCode, {
                getHighlighter: ({ theme }: { theme: Theme }) => getHighlighter({ theme, langs: [ "tsx" ]})
        } ]
]

And yes this stupid hack actually does work... I'm not calling this a production solution, just a lil hack until Vercel figures this out. https://next-base-git-tests-joshmerlino.vercel.app/shiki

clearly-outsane commented 1 year ago

It didn't work for me using app router in a turborepo either. Gonna try this instead ^