[ERR_REQUIRE_ESM]: require() of ES Module `shiki` when using `14.2.x`

[arvinxx]

Link to the code that reproduces this issue

https://github.com/lobehub/lobe-chat/tree/arvinxx-patch-1

To Reproduce

Clone Repo

git clone https://github.com/lobehub/lobe-chat

Checkout branch

git checkout arvinxx-patch-1

install deps

pnpm install / bun install

run build

npm run build

Current vs. Expected behavior

with next@14.1.4, this can be build success.

but with next@14.2.1, the build is failed. the console error:

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/shiki@1.2.1/node_modules/shiki/dist/index.mjs not supported.
Instead change the require of /Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/shiki@1.2.1/node_modules/shiki/dist/index.mjs to a dynamic import() which is available in all CommonJS modules.
    at mod.require (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/require-hook.js:65:28)
    at 13076 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/app/chat/(mobile)/mobile/page.js:1:835)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 14730 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/3251.js:11:494067)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 73251 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/3251.js:11:473537)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 83337 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/2596.js:6:28070)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 793 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/4905.js:1:36842)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 78518 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/1680.js:79:4281)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 40921 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/chunks/1680.js:215:508)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at 70080 (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/app/chat/(mobile)/mobile/page.js:6:250)
    at Object.t [as require] (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/.next/server/webpack-runtime.js:1:143)
    at require (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:16:18270)
    at I (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:94362)
    at /Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:96668
    at F._fromJSON (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:97106)
    at JSON.parse (<anonymous>)
    at O (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:94083)
    at t (/Users/arvinxx/CodeProjects/LobeHub/lobe-chat/node_modules/.pnpm/next@14.2.1_@babel+core@7.23.6_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:12:100582)
 ✓ Generating static pages (17/17)

> Export encountered errors on following paths:
        /chat/(mobile)/mobile/page: /chat/mobile
        /settings/agent/page: /settings/agent
 ELIFECYCLE  Command failed with exit code 1.

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000
  Available memory (MB): 65536
  Available CPU cores: 10
Binaries:
  Node: 20.9.0
  npm: 9.2.0
  Yarn: 1.22.19
  pnpm: 8.15.3
Relevant Packages:
  next: 14.2.1 // Latest available version is detected (14.2.1).
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.4.5
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

App Router, Module resolution (CJS / ESM, module resolving)

Which stage(s) are affected? (Select all that apply)

next build (local)

Additional context

I have check the Next.js 14.2.1-canary.3. this issue is still remain

[dev-pi2pie]

https://github.com/dev-pi2pie/next-v14_2-with-rehype-pretty-code-and-shiki-transformer

I've made a repo which reproducing the similar issue which happened on the build stage. And this repo is a simple editor, which input something like markdown syntax to the text area and get the result back.

And the issue happened in the code:

src/lib/mdsource.ts

I think the issue is from the code:

import {
  transformerNotationDiff,
  transformerNotationWordHighlight,
  transformerNotationHighlight,
} from "@shikijs/transformers";

If I do not use the transformers in the rehype-pretty-code, and the build will be success.

However, good to see @arvinxx mentioned the issue I encountered too.

[hasanshahriar32]

i have also faced similar issue. i was using nextra with app router.

this following package is creating problem from nextra: @theguild/remark-mermaid

[songtanjichuan]

me too

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

[himself65]

same error here

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

[Innei]

Currently this is the only solution, you can only import in async to ignore errors.

I'm looking at this as well.

Edit:

Top level async/await will get a warning. So we can use use hook.

Use it in async component or dynamic import this component.

const codeHighlighterPromise = (async () => {
  if (isServerSide) return
  const [{ getHighlighterCore }, getWasm, { codeHighlighter }] =
    await Promise.all([
      import('shiki/core'),
      import('shiki/wasm').then((m) => m.default),
      import('./core'),
    ])

  const core = await getHighlighterCore({
    themes: [
      import('shiki/themes/github-light.mjs'),
      import('shiki/themes/github-dark.mjs'),
    ],
    langs: [],
    loadWasm: getWasm,
  })

  return {
    codeHighlighter: core,
    fn: (o: { lang: string; attrs: string; code: string }) => {
      return codeHighlighter(core, o)
    },
  }
})()
export const ShikiHighLighter: FC<Props> = (props) => {
  const { lang: language, content: value, attrs } = props
  const codeHighlighter = use(codeHighlighterPromise)
// ...
}

[dev-pi2pie]

Good, after I got the solution from @Innei provided --> quick link

I tried this solution in the repo, which I made for reproducing this issue before. I made a new branch, and tried this solution, and it works. (with dynamic import)

The new branch in the repo for reproducing this issue: with-dynamic-import

Key diff code in src/app/page.tsx:

import { EditorProvider } from "@/components/editor/editorContext";
- import EditorResult from "@/components/editor/editorResult";
import EditorTextArea from "@/components/editor/editorTextArea";
+ import dynamic from "next/dynamic";

export default function Home() {
+  const EditorResult = dynamic(
+    () => import("@/components/editor/editorResult"),
+    { ssr: false }
+  );
  return (
    <main className="flex min-h-screen flex-col items-center justify-between p-24">
      <EditorProvider>
        <div className="lg:max-w-[1024px] mx-auto flex flex-col gap-5 mt-10 p-2">
          <h1>Editor</h1>
          <EditorTextArea className="text-black" />
          <article className="prose dark:text-white dark:prose-headings:text-white">
            <EditorResult />
          </article>
        </div>
      </EditorProvider>
    </main>
  );
}

Thanks @Innei pointed out the solution

[arvinxx]

Let me introduce a more easy way to fix it:

// next.config.mjs

const nextConfig = {
  // ... other config

  experimental: {
    optimizePackageImports: [
       // ... other packages
+      'shiki',
    ],

  // ...
}

// ...

[dev-pi2pie]

Hi @arvinxx, thanks for sharing, really a good solution, nice! 👍

I add this solution to a new branch in the repo which I made for reproducing this issue.

new branch: with-optimize-package-imports

[mpereira]

Adding "shiki" to experimental.optimizePackageImports didn't work for me. It just gave a different error message: TypeError: Cannot read properties of undefined (reading 'transformers') on codeToHtml() calls.

Adding "shiki" to transpilePackages did fix the issue for me:

next.config.js:

module.exports = {
  transpilePackages: ['components', 'shared', 'shiki'],
};

[dev-pi2pie]

Cool, @mpereira, thanks for sharing this case.

I added this solution to a new branch in the repo which I made for reproducing this issue. And also added the new component to demo what happened on while calling codeToHtml().

new branch: with-transpile-packages-and-code-to-html

[huozhi]

We had mad some module resolution improvements for ESM package on 14.2.8, the original repro branch is gone. Would someone else help check if it's working well now without the above workaround (no pkg optimization, no transpile pkg)? Thanks 🙏 If it's not working please lmk with your reproduction, I'd love to help take a look

[dev-pi2pie]

Hi @huozhi, good to let me know this good news. I've made a new branch: upgrade-to-v14_2_8 (no pkg optimization, no transpile pkg)

Work as intended, cool 🎉

[huozhi]

Amazing, thanks for confirming. I'll close this as resolved!

[himself65]

works well, thx

[FrancescoSaverioZuppichini]

Let me introduce a more easy way to fix it:

// next.config.mjs

const nextConfig = {
  // ... other config

  experimental: {
    optimizePackageImports: [
       // ... other packages
+      'shiki',
    ],

  // ...
}

// ...

this broke my loading page hahha now loading is not showing anymore!

[huozhi]

@FrancescoSaverioZuppichini could you submit a new issue with a repro? It's hard for us to tell if it's some other issue is causing that. We'd like to assist in looking into that case further

[github-actions[bot]]

This closed issue has been automatically locked because it had no new activity for 2 weeks. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.