Create a statement about migration to ESM

[mcollina]

Please read https://github.com/nodejs/node/issues/43382#issuecomment-1166267429 for context.

There seem to be quite a bit of confusion around on when is the right time to migrate to ESM for module authors.

Most of the real-world team I have worked in the last two-three years have no plan to migrate to ESM as there is no business benefit for them. Very few managers would allocate budget to the migration: it's significantly simpler to switch to a dependency that still support CJS.

The transition to ESM is taking significantly more time than what others have expected. ESM is still lacking a few fundamental features that makes CJS great. Very few companies or individuals are willing to invest the time and effort needed to fill the gap, i.e. module loaders are still in the works.

The download numbers of chalk, node-fetch and other libraries tells the same story: most users are not upgrading to ESM, as most people are staying on an old version which is a great solution short term. However, most users don't expect to migrate any of my code to ESM-only anytime soon, therefore they are seeking alternatives. A few high-impact authors decided that it was not interesting for them to support teams that have no interesting to switch to ESM: this is their decision to make. However, this decision to go ESM-only created significant issues in the community and a high number of people is asking what should they do.

What can Node.js core do? Should we recommend high-impact module authors to dual-publish? Stick to CJS?

[aduh95]

How easy would it be for someone to create a require-esm package that converts ESM to CJS on the fly (assuming no TLA)? In my experience, package consumers don't really care if a package is ESM or CJS, they only thing is they want some kind of synchronous import when their own code is in CJS.

Should we recommend high-impact module authors to dual-publish? Stick to CJS?

Node.js docs already has a long explainer on how to do it and what are the upsides/downsides of each approach. I don't think it would make sense to try to come up with an "official recommendation" – module authors have no reason to follow our recommendation if it's not backed up by facts, so instead I think we should ensure that the docs are as clear as possible on what are the consequences of such decision, and let module authors make the call.

[ljharb]

If node makes a recommendation, the recommendation for most cases should be to stick to CJS, imo, since both module systems support it. The only advantage i can think of for having a dual mode package is to provide named exports, and this can be done with a separate ESM file that imports CJS and re-exports it.

The reality, though, is that the number of authors switching to ESM-only is quite small, and i don’t think it’s reasonable to claim that these experienced developers “didn’t understand” the consequences of doing so - it seems to be a willful choice, as none of them seem to have changed their mind afterwards, even when many users complain.

Thus, i don’t think making a recommendation against ESM-only would change much, since the ecosystem has already largely decided that’s something to be avoided.

[GeoffreyBooth]

I just discovered this issue; is there a reason we didn’t ping @nodejs/modules / @nodejs/loaders ?

I helped moderate the collaborator summit session where ESM and loaders were discussed. At one point I asked the room if anyone was using ESM in their projects, and @jasnell and I were the only ones (out of about 30 or so people). But then I asked the room who would use ESM for the next project they start, and almost every hand went up. Several people commented that they viewed CommonJS as a liability; with the @sindresorhus packages and others moving to ESM-only, developers are afraid of starting new CommonJS projects that can’t use the latest versions of popular dependencies and therefore becoming stuck on old versions that become unmaintained. So while I agree there isn’t much motivation for existing projects to migrate, the transition is happening, and might even accelerate, as application developers start new projects.

As for package maintainers, I’m not sure we should be making any statement in particular, as I doubt we can reach consensus on one. For myself with regards to the CoffeeScript package that I maintain, I took the “hack the es-module-lexer algorithm” approach, with a bunch of lines like module.exports.VERSION = CoffeeScript.VERSION that es-module-lexer statically analyzes to provide named exports without me needing to write an ESM wrapper. We could add this method to the docs as a suggestion, for authors of CommonJS packages that want to keep their packages as CommonJS.

I don’t think we should be asking or encouraging package authors to not write ESM-only packages, both because many would ignore us and because such an approach goes against our other project goal of trying to improve interoperability with browsers and the other server-side JavaScript runtimes. A CommonJS package may be maximally compatible for all Node consumers, who might have either CommonJS or ESM apps, but an ESM package is maximally compatible with all runtimes, and the latter is something that is increasingly important for many authors. We could try to encourage authors to write dual-mode packages, but a) that implicitly goes against our own docs’ recommendations to avoid dual-mode and its hazards whenever possible, and b) authors would justifiably ask why they need to support CommonJS when every maintained version of Node supports ESM. I know the answer to the latter question is because many people can’t or won’t upgrade their apps to ESM, but I think authors like @sindresorhus are assuming that that won’t be the case for too much longer and it’s not worth the effort of dual-publishing during the transition period. (And publishing ESM-only hastens the end of that transition.)

So in short, I think the kids are alright; the ecosystem is fine. We need to finish closing the gap between CommonJS and ESM, and we’ve made good progress and are actually starting to get pretty close. The sooner we finish the loaders roadmap and other odds and ends missing from ESM, the sooner no one will have a reason to need to use CommonJS for a project.

[weswigham]

Most people that I've spoken to who are "using node esm" are actually using a bundler like webpack, rollup, or esbuild, are pulling in both cjs and esm modules as available, with much looser rules than what node prescribes, including optional extensions in esm usually, and when confronted with the limitations of node's esm implementation and it's poor cjs cross-interoperability become frustrated that node's functionality doesn't match what their tools can provide and expose. This comes up when, for example, someone is using esbuild for their mostly web-based build, but jest runs tests in node itself, or TS checks their code as if it were running in node directly. There's a big patchwork of tools and plugins to try to get this stuff to agree on how modules should behave, and apparently very few of them seem to think the pure node way is sufficient. I know very few people using node esm without any adornments right now - just some high profile library authors, really.

[mcollina]

I just discovered this issue; is there a reason we didn’t ping @nodejs/modules / @nodejs/loaders?

I don't think there was any specific reason on my end. This question I posed is for the TSC as it's responsible for the technical direction of the Node.js project. I should have cc'ed those teams and I apologize.

I know the answer to the latter question is because many people can’t or won’t upgrade their apps to ESM, but I think authors like @sindresorhus are assuming that that won’t be the case for too much longer and it’s not worth the effort of dual-publishing during the transition period. (And publishing ESM-only hastens the end of that transition.)

The numbers are not matching up. The latest node-fetch, chalk, and other popular libraries that went ESM-only have not seen significant adoption of their ESM version. The users are not switching en-mass, and going ESM-only is primarily causing friction in the community.

So in short, I think the kids are alright; the ecosystem is fine. We need to finish closing the gap between CommonJS and ESM, and we’ve made good progress and are actually starting to get pretty close. The sooner we finish the loaders roadmap and other odds and ends missing from ESM, the sooner no one will have a reason to need to use CommonJS for a project.

I'm glad things are progressing (I didn't know we were close!), and I agree. Frameworks and companies are already relying on the "old" loaders implementation, and I'm not sure we should break them in an LTS release. Assuming all the new loaders ship in the next few months, folks would have to wait until v16 goes out of LTS in September 2023.

[ljharb]

Even if all new apps going forward - every one - is ESM-only, that does not mean that many transitive dependencies in the tree will ever switch or feel pressure to switch to being ESM-only, and all the things those depend on will still need to support CJS.

It’s great for node to strongly recommend that apps use ESM - but it is harmful to allow users to think that packages should be ESM-only, as the last few years have objectively demonstrated.

[mcollina]

I agree with @ljharb.

[GeoffreyBooth]

I should add a few other notes from the collaboration summit. For reference, here are the notes: https://github.com/nodejs/next-10/blob/main/meetings/summit-jun-2022.md. The biggest blockers for ESM adoption on the app developer side were:

• Observability (what loaders are meant to address, and also the diagnostic channel)
• Testing (mainly Jest’s lack of full support, because it depends on the vm module which doesn’t support ESM well)
• TypeScript support, which is no longer an issue since TypeScript 4.7 but that was only released a few weeks ago and many participants weren’t aware.

Another thing that was discussed for a while was how ESM import order is different from CommonJS; in ESM the leaves of the tree are evaluated first, eventually rolling up to the original entry point; whereas CommonJS is top-down and synchronous, so the first require statement of the entry point runs first. In our discussion we landed on a recommendation, which is to make an “entry point wrapper” for anything that needs to explicitly run first, and use dynamic import statements to enforce load order:

await import('datadog') // Or whatever instrumentation/observability package(s) this app needs
await import('./startup.js') // Load polyfills, set global/environment variables, etc.
await import('./app.js') // The “actual” app entry point, with static import statements

So this is just something else to perhaps add to the docs; maybe we need a “migrating an app to ESM” section with suggestions like this, advice for solving common problems in ESM apps, etc.

[GeoffreyBooth]

The numbers are not matching up. The latest node-fetch, chalk, and other popular libraries that went ESM-only have not seen significant adoption of their ESM version. The users are not switching en-mass, and going ESM-only is primarily causing friction in the community.

I wrote a script to get the numbers of downloads per major version in the last 7 days:

chalk (went ESM-only in 5.0.0 on 2021-11-26)

5.x	3,228,382
4.x	102,789,507
3.x	18,860,621
2.x	45,461,165
1.x	21,255,066
0.x	1,555,480

node-fetch (went ESM-only in 3.0.0 on 2021-08-31)

3.x	1,365,406
2.x	48,226,176
1.x	4,162,645
0.x	13

While compared to the previous major version, the ESM-only versions have only a tiny fraction of the downloads; but 3.2 million or 1.3 million downloads per week is not insignificant by any means. There are plenty of users out there using the ESM-only versions of each package.

Also, both packages are aware of users who can’t use the ESM-only versions. They each address the issue:

• chalk: This package is now pure ESM. Please read this. . . . It’s totally fine to stay on Chalk v4. It’s been stable for years.

• node-fetch: node-fetch from v3 is an ESM-only module - you are not able to import it with require(). If you cannot switch to ESM, please use v2 which remains compatible with CommonJS. Critical bug fixes will continue to be published for v2.

[mcollina]

I'm closing this. While I still think a statement about the ESM migration from the project is needed, I prefer to spend my time on Node.js to focus on other issues rather than reach consensus on the topic. I will likely publish my point of view in a blog post.

[ljharb]

Absolute numbers of packages are not relevant, due to automated mechanisms caching downloads - only relative numbers are useful. As such, the thing to note is that for chalk, v5 represents only about 3% of v4's downloads, let alone v1-3; and for node-fetch, v3 is also only about 3% of v2's downloads, let alone v1.

The extremes here imply that less than 3% of potential users for each package are doing some combination of a) starting a new ESM project, or b) migrating a CJS app/package to ESM.

While this may or may not represent "plenty" of users, it certainly does not represent a significant trend, especially after nearly a year.

If you look at other packages by these authors that have gone ESM-only, the picture painted is even starker.

[Trott]

I would be cautious about inferring too much about ESM adoption from the major version bumps of these packages. The impact of going ESM-only may be difficult to distinguish from the friction of a major-version bump, especially if Node.js import works fine with the previous CommonJS version, as it does in most cases it seems. You can see the challenge when you look at download stats for some things that aren't ESM-only but have had major version bumps. For the trim package, version 1.x has 2.5% of the downloads of version 0.0.1, comparable to the 3% figure for chalk and node-fetch. But version 1.x is a CommonJS module, so there's something else going on here, not ESM avoidance. (And version 0.0.1 has a high-severity audit warning, so you'd think people would be very motivated to move.) Part of it for trim is that 0.0.1 is 9 years old and it just stayed there for 7 years before another version was published. Another part of it is that people may not be updating but instead (wisely) migrating to String.prototype.trim() instead. But all of this is not beside the point. It's easy to infer specific causes for these numbers, but it's much more difficult (impossible?) to understand the relative magnitudes of the different causes.

All that said, I do think there's ample reason to believe that ESM adoption is less than what many of us had hoped for, and that ESM-only modules introduce friction/hardship for users. (That may be in exchange for easing the burden on maintainers, though. I'm not making any judgments here myself either way. But pain for Node.js end users OR module maintainers is something we shouldn't completely ignore. Just not sure what a proper statement or other response here would actually be, especially given the heterogeneous nature of Node.js maintainers, etc.)

[JakobJingleheimer]

My 2¢ since I was tagged:

Most of the real-world team I have worked in the last two-three years have no plan to migrate to ESM

I have no idea what this is talking about. I haven't seen anyone write CommonJS in nigh a decade, nor publish new apps or libraries in CommonJS in as long. There is very clear business value, which is why everyone is doing it. Libraries may not be publishing ESM-only (although I anecdotally don't believe they aren't, and every library maintainer I've spoken to / has reached out to me has said almost verbatim that CJS is a nuisance), but consumers absolutely are leaving CJS in droves. The temperature of the room is CommonJS is legacy.

The transition to ESM is taking significantly more time than what others have expected.

As far as I've seen, this is the nail on the head. People are chomping at the bit, and the gate just won't go up: they're blocked by some dependency that is itself blocked by some specific feature they can't get in ESM yet. Consumers are fed up and abandoning those packages wherever they can.

If I may back-pat my team a bit: I think we have made significant progress in the past year. I think we are about ½ way there and have clear designs for the second ½, and the capability and momentum to get there.

I dunno what happened with Qix or Chalk (the cited post sounds like half the problem was someone from Node straight-up plagiarised his work?).

Regarding tracking packages' ESM versions via download numbers, that is basically completely invalid, relative or not: there are way too many lurking variables. Most obviously, a decade of old CJS-era applications getting re-deployed with minor changes. I know of at least a dozen enterprise examples doing this, skewing the numbers. These would of course disproportionately skew because of the years they had to accumulate and the lack of sophistication available (unrelated to CJS), whereas modern/ESM-era apps etc are highly optimised to minimally impact these numbers (further skewing the numbers against them).

Regarding a recommendation to library authors, there are a couple very viable interop options (basically, distribute specific CJS). I think libraries should do this based on their own consumers usage as a transition period, and then cut over to ESM, as the industry trend is starkly moving that way. And this is what I've anecdotally seen happening anyway.

I think we should make an official stance/recommendation rather than leave the community in disarray. The longer we wait, the less our voice will be hear.

[GeoffreyBooth]

I dunno what happened with Qix or Chalk (the cited post sounds like half the problem was someone from Node straight-up plagiarised his work?).

This is discussed in https://github.com/nodejs/node/issues/43382. Basically yes, Qix is upset that Node is considering “stealing” his library. He suspected that one of the motivations was that it went ESM-only, which after his complaint we unfortunately confirmed, which then spawned this thread; but there were other, in my opinion stronger (and less controversial) reasons for wanting Chalk or an API like it to move into core. Even if he were to turn around tomorrow and start dual-publishing Chalk, I think there would still be interest in moving it into core, so I think we should consider Chalk a separate issue.

The transition to ESM is taking significantly more time than what others have expected.

One way to look at things is that the goal of the ESM-only package authors is being achieved, if my anecdote from the summit (where everyone said that their next project will be an ESM one, to avoid being left behind as dependencies all move to ESM-only) is to be assumed to be representative. Those package authors want to drag the community across the transition into ESM, and they’re using the carrot (or stick) of “want the latest version of my package? Go ESM” to do it. And the more that people start to migrate, the more pressure then builds on the laggards of the ecosystem (Jest, observability tools, build tools, formerly TypeScript) to better support ESM too. From the perspective of the authors, this is all a good thing, and the full migration won’t happen until the end users demand it of their tools—it was a chicken-and-egg problem until the package authors started forcing the issue. And this pressure includes Node itself too; Jest and observability tools can’t provide fully equivalent functionality in ESM as they do in CommonJS right now because of Node itself, so if those tools feel pressure to support ESM then there’s also pressure on Node to improve support (which we could then leverage to ask folks from those teams to help us out with contributions). From this perspective, actually completing the migration is the goal, and the friction that some end users are feeling right now is the stick that actually makes it happen. (How long did people stick around on Python 2 before the Python team forced the issue by dropping support?)

I think we should make an official stance/recommendation

I think first we need to clearly identify the problem we’re aiming to solve. What exactly is the friction or frustration, and on whose part, that we want to address? What are all the potential solutions for it? It might be that a statement, especially one that gets ignored, is not the best solution. It might also be that maybe making life too easy for CommonJS diehards is not in the project’s long-term interest, as we get increasing competition from server-side runtimes that are ESM-only (and package authors who want those runtimes’ users increasingly develop ESM-only packages).

If the problem we’re trying to solve is “app developers writing CommonJS apps are apprehensive about feeling left behind as their preferred dependencies drop support for CommonJS,” I don’t think any amount of public statements or cajoling will convince enough package authors to change their ways—and it will make us look bad, as if we’re criticizing our own work or abandoning the ESM effort. Also, crowdsourcing the effort of dual-publishing ESM-only packages is incredibly inefficient; rather than asking every package author to do more work, someone could automate the process. Someone could make new packages on the npm registry called chalk-cjs and node-fetch-cjs and so on, and on every publish of a new version of chalk a CommonJS-transpiled version gets published on chalk-cjs etc. It would be the npm registry equivalent of unpkg.com providing automatically ESM-transpiled versions-,%3Fmodule,-Expands%20all%20%E2%80%9Cbare) of all npm packages. This could just as easily solve the problem of CommonJS users not feeling left behind, without antagonizing package authors.

Anyway I’m getting ahead of myself. My point is just that if we want to do something about this concern, let’s start a new discussion issue about it and start from the beginning, and we can see what all the ideas people have are. Maybe we have a technical solution and some kind of recommendation or instructions in the docs or whatever. We should explore all our options, including weighing their burdens—who are we asking to do work, and why—and pros and cons including the long-term goals and health of the Node.js project itself.

[jasnell]

Sidebar:

Basically yes, Qix is upset that Node is considering “stealing” his library.

Let's be clear: this was the accusation that was made but is not the reality. No one attempted to plagiarize or steal anything.

[ljharb]

package authors started forcing the issue

I think this is the problem right here. If there's a new X (module system, in this case), and it's not "better enough" for people to WANT to migrate to it on their own, then it's not better. The ecosystem shouldn't ever be forced, and won't be forced, and the end result is just going to be that new packages that support CJS (and thus, both module systems) end up taking over from the ones that try to manipulate users into making a different choice.

[silverwind]

Has there been any thoughts about the deprecation of CJS? Starting with docs-only surely would nudge a few people into ESM adoption, which imho is far too slow and at the current pace, would probably take another 10-20 years to complete while other runtimes that support only ESM are sprawling.

[ljharb]

If something's better, it will get adoption on its own merits - it doesn't need a nudge.