Migration to ES Modules

[Josh-Cena]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

Motivation

I did a very naïve attempt to migrate to ESM in #5816, but I far underestimated the difficulty in pushing it through. After some pondering, I think this should be rolled out progressively.

Status of ESM

ESM is a new type of Node modules system, replacing the old common JS system (require + module.exports). For the engine, the parsing goal is different ("module" or "script"), so a file needs to be determined as ESM or CJS before executing it, either through the .mjs extension or through the "type": "module" entry in the nearest package.json.

If a file is ESM, it can import from other ES modules. However, it may not be able to import all CJS modules, depending on how the CJS module is structured, because exported symbols, per the ES spec, need to be lexically determined, while CJS can be far more dynamic.

If a file is CJS, it can't import ESM modules, unless the await import() dynamic import is used. However, this is against the norm of how most modules are imported. This effectively means as soon as a considerable number of the dependencies are ESM, we have to migrate ourselves.

Many packages on NPM are now ESM-only, most notably the packages by Sindre Sorhus, and MDX v2, which is the pillarpost of our architecture.

Benefits

1. Unlock future dependency upgrades. Many popular libraries are seeking to upgrade to ESM; if we can be ESM, we can interoperate with them. For example, MDX v2, chalk...
2. A similar transpilation target for client and server code; no need for multiple tsconfigs
3. Permit top-level awaits

Blockers

1. TypeScript has very lame support of ESM transpilation so far. It seems their deferred ESM support still won't land in 4.6, so in the meantime we may have to run all our Node code with --experimental-specifier-resolution=node and keep the old resolution
2. Jest doesn't seem to like importing ESM dependencies
3. We use import-fresh to bypass cache and always import fresh modules for hot reloading, etc. But ES Modules don't expose caching manipulation yet
4. Community dividing: although the ESM migration will surely be a major version, it means in the foreseeable future after that, some plugins will not be compatible with the new version (especially those that import utils and logger)

Actions needed

1. Removing __filename and __dirname. These globals don't exist in the ESM scope, replaced by the import.meta.url
2. Setting target: 'nodenext' in the tsconfig.
3. Changing our import paths. ESM requires the index.js name and the .js extension to be explicit.
4. Setting "type": "module" in our package.json.
5. Tweaking the configuration for related tools?

Plan

1. Config files: JS config files like docusaurus.config.js and sidebars.js can be allowed in ESM once we figure out how to bypass cache and fresh-import ESM
2. Utils: this includes utils, utils-validation, logger. They should always be distributed as dual-package because plugin authors are likely to import them as CJS.
3. Core: the biggest blocker is still the extensive use of import-fresh. Migration to ESM means we can import both CJS and ESM plugin modules with await import. However, because users only interact with the core through its own CLI, we don't have to care about others importing this.
4. Plugins: migration of plugins can only happen after migration of core (or at least solving import-fresh in the core), because they have to be imported as ESM.

Related issues/PRs

If an issue or PR is related to the ESM migration process, please link to this meta-issue and we will add it to the list below for tracking purposes.

• 5379

• 5816

• 6286

• 6521

• 6661

• 6716

• 6898

• 6899

• 6921

• 7371

• 7379

[iPurpl3x]

Yes !!!! +1 ! Finally I stumbled on this issue, I had updated @mdx-js/react and had errors and it took me a while to understand what happened...

Error was something like

_mdx_js_react__WEBPACK_IMPORTED_MODULE_2__.mdx is not a function

and

export 'mdx' (imported as 'mdx') was not found in '@mdx-js/react' (possible exports: MDXContext, MDXProvider, useMDXComponents, withMDXComponents)

[Josh-Cena]

@iPurpl3x Your problem is probably not because of ESM, because @mdx-js/react is used on client-side, where Webpack can take care of ESM syntax. Instead, it's because of the API changes in v2. MDX will be compiled to a JSX file containing a line import { mdx } from '@mdx-js/react'; which doesn't exist in the latest MDX version.

[Josh-Cena]

I ran yarn outdated and below are all dependencies that are ESM and we can't upgrade:

Outdated dependencies

``` Package Current Wanted Latest @mdx-js/mdx 1.6.22 1.6.22 2.0.0 @mdx-js/react 1.6.22 1.6.22 2.0.0 boxen 5.1.2 5.1.2 6.2.1 chalk 4.1.2 4.1.2 5.0.0 escape-string-regexp 4.0.0 4.0.0 5.0.0 globby 11.1.0 11.1.0 13.1.1 hast-util-to-string 1.0.4 1.0.4 2.0.0 is-root 2.1.0 2.1.0 3.0.0 leven 3.1.0 3.1.0 4.0.0 mdast-util-to-string 2.0.0 2.0.0 3.1.0 rehype-parse 7.0.1 7.0.1 8.0.4 remark 12.0.1 12.0.1 14.0.2 remark-emoji 2.2.0 2.2.0 3.0.2 remark-math 3.0.1 3.0.1 5.1.1 remark-mdx 1.6.22 1.6.22 2.0.0 remark-parse 8.0.3 8.0.3 10.0.1 remark-stringify 8.1.1 8.1.1 10.0.2 stringify-object 3.3.0 3.3.0 4.0.1 to-vfile 6.1.0 6.1.0 7.2.3 unified 9.2.2 9.2.2 10.1.1 unist-builder 2.0.3 2.0.3 3.0.0 unist-util-remove-position 3.0.0 3.0.0 4.0.1 unist-util-visit 2.0.3 2.0.3 4.1.0 ```

Mostly Sindre Sorhus packages and MDX ones

[sachaw]

This is just going to get worse with time, @Josh-Cena Thanks for the great work, hopefully we can get Docusaurus using ESM soon.

[sambacha]

sindre is working on converting his packages to pure ESM, see https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

[slorber]

also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

What do you mean by "does not work"? How do you run it and how do you see it not working exactly? It seems to work for me

[Josh-Cena]

@slorber It was a mistake made in the last publish😅 The file is called bin/index.mjs but in package.json it's referred to as bin/index.js. You probably need to run it outside our workspace. See #6897

[Josh-Cena]

TS 4.7 seems promising: https://devblogs.microsoft.com/typescript/announcing-typescript-4-7-beta/#ecmascript-module-support-in-node-js We should be able to investigate after that

[slorber]

Yes, also great to have "exports" field support!

[arobbins]

This might be a silly question, but If I'm creating a React library that I want to import into Docusaurus, what is my workaround at the moment? Do I just need to build the library as CJS and not ESM?

[Josh-Cena]

@arobbins If you are importing client-side code (mostly theme components) from Docusaurus, we are always distributing them in ESM format because it can be handled by Webpack. You can use either format.

[webbertakken]

Currently I'm missing the following in the benefits:

Better developer experience through not having to compile with babel, making for an overall faster dev server performance.

[Josh-Cena]

I'm still unsure about how ES Modules would play out in the client side. The plan above is more for Node-side stuff. If we manage to ship ES Modules in dev mode (like snowpack does), that would definitely be a gain as well!

[webbertakken]

Exactly. Please note that Snowpack is no longer maintained, but Vite has become quite successful.

[Josh-Cena]

Yup! We have already faced some issues with the client source files being pure ESM (https://github.com/facebook/docusaurus/pull/7379 and https://github.com/facebook/docusaurus/issues/7238), so we probably have to get the dependencies straight before thinking about the migration...

[Josh-Cena]

Yarn PnP has an issue with chalk v5's use of package.json import fields: https://github.com/yarnpkg/berry/issues/3843 Our ESM migration is on hold because of this.

[mrienstra]

Yarn PnP has an issue with chalk v5's use of package.json import fields: yarnpkg/berry#3843 Our ESM migration is on hold because of this.

@Josh-Cena, thoughts on other options, so ESM migration can proceed? I don't see any movement on yarnpkg/berry#3843, and as you probably already know, chalk/chalk#531 says "This is a problem with Yarn PnP and should be reported on the Yarn issue tracker.", so it's not going to get fixed on that end.

Is moving from yarn PnP to pnpm an option? Looks like doing so would require moving from dependabot to renovatebot (see dependabot/dependabot-core#1736).

In the unlikely event it's helpful, some yarn-->pnpm migration PRs: Vue: vuejs/core/pull/4766 Next.js: vercel/next.js/pull/37259 Vite: vitejs/vite/pull/5060 Pinia: vuejs/pinia/pull/1179 Browserlist: browserslist/browserslist/commit/6d0c552 (couldn't find a PR for this one)

[Josh-Cena]

We are not using PnP. I'm saying migrating to ESM will break all users on PnP, which we are not going to do because we have E2E tests running on PnP and we are committed to support them.

[mrienstra]

Gotcha, I see that now, thanks for explaining it to me.

[thomasmattheussen]

https://github.com/yarnpkg/berry/issues/3843 seems to have been fixed yesterday!

[mrienstra]

yarnpkg/berry#3843 seems to have been fixed yesterday!

Nice! Looks like that change is available in 4.0.0-rc.22. Run yarn set version canary if you want to try it. v4 release is still a little ways off ("may take a couple more months"), but:

[...] what's in master is stable, and I'd recommend you to try it. The only notable difference with stable is that we reserve the right to land a couple more breaking changes in future RCs, but in terms of stability it's almost always better to use RCs than stable.

... according to https://github.com/yarnpkg/berry/discussions/4895

Edit: See https://github.com/yarnpkg/berry/issues/3591 for v4 breaking changes. I've seen two mentions of people stumbling over enableGlobalCache default changing from false to true. (Google search: site:yarnpkg.com "enableGlobalCache")

[mrienstra]

Update: the fix that adds support for the package.json includes field is also in the Yarn 3.2.4 release.

[zfm-alyssonteixeira]

I've a problem trying to do npm install in docker-compose on my docusaurus, that has nothing to do with this, right? Stack Overflow docusaurus doesn't work with docker npm install

[NickGerleman]

Edit: moved to https://github.com/facebook/docusaurus/discussions/9435#discussioncomment-7344457

---

Something I ran into when converting a project to ES Modules was that BrowserOnly and importing code for the browser, but not the server, no longer works as documented.

In CommonJS, browser specific code can be dynamic required synchronously inline with the component provider. But this is async with ESModules, so the existing example is no longer correct.

I ended up wrapping the component in React.lazy, and adding suspense, which works, but is a little unergonomic, and has different semantics (since the page now renders before the browser only code is available).

[slorber]

Edit: moved to https://github.com/facebook/docusaurus/discussions/9435#discussioncomment-7344457

---

@NickGerleman this issue is more about using ESM internally inside Docusaurus, and not really about using ES modules in docs and React code with <BrowserOnly>.

In CommonJS, browser specific code can be dynamic required synchronously inline with the component provider. But this is async with ESModules, so the existing example is no longer correct.

I assume you are on v3 (React-Native website?). Can you please open a separate issue with the code that you are using, and the tell me which existing example is now incorrect?

I ended up wrapping the component in React.lazy, and adding suspense, which works, but is a little unergonomic, and has different semantics (since the page now renders before the browser only code is available).

That works but in the v3 release notes (now available with the fresh RC.0) I added that this is experimental because it's possible we'll have to change the semantics later. https://docusaurus.io/docs/next/migration/v3#react-v180

[NickGerleman]

Edit: moved to https://github.com/facebook/docusaurus/discussions/9435#discussioncomment-7344457

---

@slorber this is for an effort to port https://yogalayout.com/ (currently using an ancient version of Gatsby) to Docusaurus. Right now I have it targeting 2.4.

The website has an interactive component, using an asm.js version of Yoga. In modern Yoga, the JavaScript bindings are WebAssembly instead. The current iteration of WebAssembly best practices interoping with bundlers requires top-level await support, which forces the component to be an ES Module.

Wholesale porting the existing component led to a non-descriptive error message about server compilation failing. Originally I knew this was due to incompatibility with the WASM builds, but I think now it might be a different incompatibility I haven't been able to suss out (I suspect one of the very old dependencies being pulled in). In the meantime, I have been running this playground only on the client.

Right now, the documented way to load a module only in the browser looks like this:

    <BrowserOnly fallback={<div>Loading...</div>}>
      {() => {
        const LibComponent = require('some-lib');
        return <LibComponent {...props} />;
      }}
    </BrowserOnly>

If the component is an ESModule, we cannot synchronously require it, but we must return a component synchronously in the API.

    // We can't use this pattern
    <BrowserOnly fallback={<div>Loading...</div>}>
      {() => {
        const LibComponent = await import('some-lib');
        return <LibComponent {...props} />;
      }}
    </BrowserOnly>

We cannot use a normal import, or top level await, outside the BrowserOnly, since it would then be imported on the server as well.

We can wrap the import in React.lazy to create a sync looking React component that suspends until the import is complete, but this will happen after the module has already loaded, so we need to explicitly handle async state.

[samos123]

I hit an error after migrating to ES modules due to generation of client-modules.js which uses require: https://github.com/facebook/docusaurus/blob/7dcad0c6322c87131f9f39cbc4e765b0e1119fe8/packages/docusaurus/src/server/index.ts#L182

The V3 announcement made me think I should be switching to modules. I might be doing something silly, not a JS expert.

[slorber]

@samos123 this issue is about running ES modules natively inside the Node.js runtime (or publishing ESM package), not about public-facing features supporting ESM.

We already support ESM syntax in client modules (our own site use it for a while if you look for an example), and v3 brings ESM support for site config, sidebars etc...

If you encounter a bug, please open a dedicated issue. "I hit an error" without any extra information is not going to help us in any way help you.

[Zamiell]

homotechsual from the Discord server told me to post this bug report here, sorry if this is off-topic.

Docusaurus does not seem to support the "type" field in the "package.json" file. Subsequently, it seems impossible to use ESM with Docusaurus. (To be clear, you can have an ESM config file, but the project itself can obviously not be ESM.)

Here are steps to showcase the problem:

# Create a new Docusaurus website. This line is copy-pasted from the documentation here:
# https://docusaurus.io/docs/typescript-support
npx create-docusaurus@latest my-website classic --typescript

# Go into the website.
cd my-website

# Build the website and watch it succeed.
npm run build

# Edit the "package.json" file and add: `"type": "module",`
vim package.json

# Build the website and watch it fail.
npm run build

# Edit the "package.json" file and change "module" to "commonjs".
vim package.json

# Build the website and watch it fail again.
npm run build

This seems very unexpected, as it was my understanding that the type field defaults to "commonjs", so omitting it entirely should be equivalent to putting "type": "commonjs", but I guess that isn't the case here.

1) Should Docusaurus be fixed such that "type": "commonjs" is equivalent to omitting it entirely? 2) Is it intended that ESM is not supported in the latest version of Docusaurus? 3) Is this issue about updating the Docusaurus monorepo itself to ESM, or is it about end-user projects using ESM, or both?

[Josh-Cena]

@Zamiell type: commonjs is not equivalent for Webpack. When type is absent, Webpack allows both ESM and CJS syntax in the same file; when the module type is unambiguous, it only allows one.

[amanaknows]