.mjs extension trade-offs revision

[YurySolovyov]

I obviously can't speak for the whole community, but it seems like a lot of people are not happy with .mjs.

One of the main arguments to keep .js is that if we can detect 99% of cases where we CAN tell if is it CJS or ESM (or where we just know what to do), we may just call rest 1% edge cases and deal with it.

We can even come up with some linter rules and/or workarounds to simply teach people to do the right thing.

[bellbind]

import statements should not allow with ?type=module query, also require() function not allow ?type=nomodule query. Usual import only parse as module style js, not as traditonal script style js.

When the case:

// the module.js has no`import` and `export` statements.
import "./module.js";
import "./module.js?type=nomodule";

It should load "module.js" and "module.js?type=nomodule" independently because they are different resource.

[bmeck]

@bellbind that would not sync between web and node.

[bellbind]

I thought ".mjs" matter is incompatibility for browser JavaScript (also bundler/transpilers), that is only file extension difference. It is an idea for keep the ".js" extension but introduce difference between script js and module js (also compliant with ECMAScript module specification).

[bmeck]

@bellbind nope, .mjs is browser safe. It is even in the examples! This is because browsers don't do filename sniffing.

[bellbind]

I see. Were there rejected proposals/discussions like extending within path strings previously?

[bmeck]

No, they generally involved content sniffing, package.json fields, or file extension (kinda pathing on that one).

[bellbind]

Thank you @bmeck.

[pflannery]

Is it not possible to just distinguish by the loader used?

Here my current thinking on how a parser could interpret the loader

  // package.json
  "main" : "foo.js" // default for common js 
  ...
  "main": { // for esmodules something like this
    "path": 'index.js',
    "loader": 'esmodule' // and 'common' is always default when omitted
  }
  ...

• only parse common js when loaded via require()
• only parse esmodules when loaded via import and import()
• esmodules can still obtain common js via require()
• if desired, common js modules could still obtain esmodules with import() and maybe even have an improvised importSync().

I personally think nodejs require() should be phased out and replaced by esmodules.

[ljharb]

@pflannery that wouldn't work for the multiple (and important) use cases that involve no package.json whatsoever. Separately, changing from CJS to ESM, or ESM to CJS, shouldn't affect consumers - meaning you need to be able to require ESM, and import CJS, seamlessly.

[pflannery]

@ljharb Thanks for the reply

I agree it needs to be seamless. I think most will agree it's far from seamless if someone has to change the file extension in order to execute ESM. Also, having two different loader spec trying to load each other is complex and will be confusing for consumers. That's why I think keeping import and require separate is the cleaner way to go because both are well documented. Plus there is the added benefit that both can be made functionally available to each other (excluding any esm syntax).

I don't know all the no package.json cases but I do know that the ones that exist today are using common js and are being executed like node script.js so a possible solution for consumers migrating to esmodules is they could have a flag like node -l esm script.js

[hax]

1. Can we do some investigation on current npm packages to check how many ambiguous cases in real life?
2. Though dynamic import() is valid in Script/CJS, no one use it in real CJS up to now, and it report parse error up to now. So we can make it only work in node.js Module and exclude it from CJS.
3. Currently we need to parse twice to give a guess, but I think it's just limitation of current parser, it should possible to parse just once and convert the AST as Module/Script/CJS, just like allow return in global.

[ljharb]

@pflannery You aren't "changing the file extension" for ESM, because ESM doesn't exist yet. You'll simply create the file with the proper file extension in the first place - quite seamless.

Having two different parsing goals (already decided, and in the language spec) is what might be complex and confusing; node's choice of implementation detail has no impact on this.

Plus there is the added benefit that both can be made functionally available to each other (excluding any esm syntax).

How would you propose being able to bring in an ES module, in a CJS script, if not with require?

@hax import() is explicitly designed to work in both Script and Module goals, so in fact it would be violating the spec to disallow it in CJS modules. That's not up to node.

Again, there exist nonzero cases where something would parse as both a Script and a Module, and only actually executing the code and observing the behavior could potentially (not even with certainty) tell you if the parsing goal made it behave differently.

[pflannery]

How would you propose being able to bring in an ES module, in a CJS script, if not with require?

via dynamic import which isn't too far away from stage 4

[ljharb]

@pflannery fair enough. However, that wouldn't achieve the goal of being able to refactor from CJS to ESM without forcing consumers to update their code.

[pflannery]

@ljharb could .mjs files be optional and still be able to execute .js files using something like node -l esm script.js?

[ljharb]

A --module flag seems like something reasonable imo; that wouldn't apply to requires/imports tho - just the entry point.

[hax]

@ljharb I understand import() should work in both Script and Module, but it's mainly for the Web, which need a mechanism to import modules in scripts. And CJS is not a normal Script without any module mechanism, CJS already have require(). In practice, there is no need to use both import() and require(), and up to now there is no such code. So I think it's ok to:

• Disallow import() in all legacy CJS module, and
• Only allow import() in ESM

So that we can treat import() as the sign of ESM, just like import and export declaration.

[ljharb]

@hax a) it's not mainly for the web, node requires conditional and dynamic imports as well; b) quite a lot of node/npm code is for the web, and node can't ignore that. Regardless, disallowing import() in CJS modules would be a spec violation, since it's required to work everywhere - neither node or v8 will do that.

[hax]

@ljharb

1. require() already could do conditional and dynamic load.
2. Though there are many npm code for the Web, none is use import() up to now. Even some use it, they are rely on babel/rollup/webpack transformations which will never break Node.
3. I don't think disallowing import() in CJS is spec violation, just like we don't think allow return in global in CJS is spec violation.
4. In fact, allow both import() and require() in one script is confusing and may have many subtle semantic issues both for implementation and developers' undestanding.

[ljharb]

1. Yes, but it's not syntactic.
2. it will break node once browsers ship it, and node doesn't support it.
3. Happy to discuss with you off-thread; but disallowing import() in CJS absolutely would be a spec violation. Let's not debate it further here though.
4. disagree, but clearly this is subjective.

[hax]

@ljharb Sorry for the debate. Let me say it in another way.

I don't propose treat import() invalid in CJS, but propose treat import() as a sign of ESM if there is no other mechanism to distinguish. If there is other mechanism to denote it's a CJS, leave the import()/require() question to the developers and tools like eslint.

[WebReflection]

none is use import() up to now. Even some use it, they are rely on babel/rollup/webpack transformations which will never break Node.

... about that: https://github.com/WebReflection/import.js#importjs

[flying-sheep]

hi, what about my .js proposal?

i call it the escalator approach, as it has two separate paths that each go in only one of both directions:

• import()/import for importing ES6 modules
• require for importing CJS modules

Forward Path

A path for CJS to being using ESM modules must exist.

• import (the statement) can only import ES6 modules.
• import(path) will return a promise that resolves if/when path can be imported as ES6 module. Can also only import ES6 modules.

• It must be possible for CJS files to import() ESM.

✓

Backwards Path

A path for ESM to be created that uses legacy or CJS files must exist.

• import require from 'node' or import require from 'commonjs'
  (also solves how to provide require.resolve to ES6 modules)
• require() can only import cjs modules

• It must be possible to import CJS files.
• There must be an upgrade path for ESM to be safe against CJS becoming ESM.

✓✓

• Mixed mode situations (both ESM and CJS in same app/package) must be supported.

package.json contains module and/or main fields. the module field points to the root for ES6 modules. e.g. require('deep/path') and import deepPath from 'deep/path' works for this package:

├── package.json    # { ..., "module": "mod", "main": "index.js" }
├── index.js        # cjs
├── deep
│   └── path.js     # cjs
└── mod
    ├── index.js    # es6
    └── deep        # es6
        └── path.js # es6

ESM only future

• It should be possible for the ecosystem to move to be ESM only for newly written code.
• Whatever path is taken, it should be considered debt if files are still easily or accidentally able to be CJS.
• Whatever path is taken, it should be ready for the so called "3rd goal" problem.

✓✓✓

[YurySolovyov]

Won't that require a lot of re-wrapping of CJS to made them work with ESM?

[flying-sheep]

i think i don’t understand what you mean. could you please elaborate?

• upgrading your CJS package to a hybrid package means to add a directory (or file) for the ESM and the module field to the package.json
• every module importing your package using require can still do it.
• your ESM can use every existing CJS module via import require from 'commonjs'; const foo = require('foo')

[YurySolovyov]

I mean how do you import CJS via import statement? import path from 'path'; from node's core

[flying-sheep]

you don’t, that’s the point of this proposal. it’s mentioned in the first 4 lines, and repeated below.

that’s no problem:

• bundlers still have the same import → require semantics
• the .mjs “only default export” semantics would also require changes in ESM code that used to do import { fun } from 'some-cjs-module'

[YurySolovyov]

but people will want to use consistent constructs to import stuff in a uniform way

[flying-sheep]

explicit is better than implicit.

if people want that, they should ask maintainers for CJS-only packages to upgrade to hybrid packages until they don’t have to use require anymore.

[YurySolovyov]

What about node itself?

[flying-sheep]

hybrid packages.

[flying-sheep]

but people will want to use consistent constructs to import stuff in a uniform way

and btw: my proposal has consistent constructs to import stuff in uniform ways:

• import/import() to import ES6 modules
• require to import CJS modules

consistent, uniform, explicit.

[ljharb]

@flying-sheep your proposal means that I can't refactor a CJS module to be an ESM module without requiring consumers to change how they import it. I must be able to require ESM, and import CJS, transparently. "Hybrid packages" is not a complete solution to this (just like anything involving package.json), because not everything is in a package.

[jkrems]

I must be able to require ESM.

I don't think that's a thing anymore. All work right now goes into async loading and that will not support requiring ESM.

I personally also have to say that implicitly switching between CJS and ESM sounds rather weird. If we implicitly import CJS modules and transparently switch over to ESM, we'd force module authors to treat adding ESM support as a breaking change. Because consuming modules would suddenly get a different interface (unless your module keeps exporting the same exact default export).*

(*) Even then I wouldn't be 100% certain if it might not break completely valid uses of your old interface.

[mikeal]

I personally also have to say that implicitly switching between CJS and ESM sounds rather weird.

You're thinking about this from an implementation perspective, which is not the perspective of the vast majority of users.

Users want to import/require "packages" and they CANNOT be required to understand the underlying implementation details in order to import/require them. There are far too many packages used in the average application to burden the user with knowledge of the implementation details of each package.

[bmeck]

@jkrems correct on the CJS->ESM transition being an interesting thing; however, you can keep your default export as the CJS API and provide named exports after the transition (you could even make it so that export * from './cjs.js'; if you want to keep CJS untouched).

The goal is to provide a path for new code to be written in the standard syntax/language of ESM. Meaning:

• require('foo') has some way to work regardless if foo is ESM or CJS
  • achieved by the foo package author by shipping both .js and .mjs to the resolved specifier.
• CJS needs a mechanism to import regardless of target mode
  • achieved via import() being in both ESM and CJS. this has an affect of using the new URL based resolution. Since no modules are using real ESM import(), this is not a backwards compat concern for package consumers. It does however mean that package authors need to be mindful of breaking changes if they rely on differing resolution such as NODE_PATH.
  • support for require(ESM) has some oddities and I'd rather we keep that off the table until we see if only having import() works.
• ESM needs a mechanism to import other modules (.js/.json/.node/etc.)
  • achieved via the default = module.exports facade

[jkrems]

You're thinking about this from an implementation perspective, which is not the perspective of the vast majority of users.

I'm thinking about it from the perspective of "We (Groupon) have 100+ apps, each with hundreds or thousands of total dependencies. Wonder which one of these will mess up and cause our node X upgrade to be delayed by weeks until it gets sorted out." So, definitely consumer perspective. The implementation side is pretty straight-forward.

[bmeck]

@jkrems can you clarify with a concrete example. I would like to keep in mind this is largely one direction import being able to support modules unable to transition to ESM (like .node addons). By having require not able to load ESM it should fail pretty loudly on code bases using require which is what everything transpiles to today.

[jkrems]

After thinking it through, I think I missed the "gap-less import chain". E.g. it could only break on a node upgrade if your code is also changing to importing at least one thing. Otherwise you'll only hit require (and will get all the same code as before, as expected). 👍 Sorry, my bad. :)

[unional]

I have been following the conversation loosely and can see the frustration from both angles. Maybe one crazy solution is finding a node major version and dump the interop and mandate a wrapper module to export CJS in ESM. This way the consuming module can be used in both node and browser when ESM is natively supported.

[ljharb]

@unional the issue isn't "how not to do a breaking change", it's how to encourage adoption. If the cost of both consuming and authoring ESM is too high, when weighed together, the ecosystem simply won't migrate to it, and it'll be DOA.

[unional]

Yes, but that only applies to node. When looking at JS ecosystem at a whole (node and browser, JS/TS/Coffee), the adoption of ESM is much more likely.

[ljharb]

All the tooling that ships code to browsers uses node; including linters, minifiers, bundlers, transpilers, etc. node has a very wide reaching impact.

[unional]

I agree that nodejs has a very wide impact, and is one of the main driving force of the community. At the same time, browser support is also another main driving force. Adding things that will twist the arms of the browser vendors will likely be a tough battle and cause damage to the community as a whole.

All I'm saying is beware of not becoming the new IE when making decisions. 🌷

[unional]

Another trade-off of introducing .mjs: Browser may push to requiring of specifying the full relative path: import x from './somedir/x.js' Introducing .mjs could make the adoption harder, and in a sense defeating the purpose of interop:

The purpose of interop is to empower the consumer to use ESM/CJS code in a similar fashion, optimally do not need to be aware of the differences.

That's why we propose to do import cjs from 'some-cjs' instead of const cjs = require('some-cjs').

If we have to specify the file extension in the future and there is a .js and .mjs, then the consumer have to do either import x from './x.js' or import x from './x.mjs', and the "sameness" is gone. 🌷

[ljharb]

@unional whatever the browser requires, transpilers and/or server rewrite rules can handle it transparently (ie, can change import x from './x' so it has the proper extension in the output) - so that's a nonissue, unless you're laboring under the delusion that it'll ever be practical to do web dev without a build process.

[unional]

Sure, understand that. TS also has to face the same problem.

However, it does bring one question: what and when to build.

Should we be "building" when publishing? or the build process should be handled straightly by the end consuming application? i.e., from the TS perspective and browser-spec perspective, the node-consuming-bits and the browser-consuming-bits can be all different by .js vs .mjs. And the npm and other toolings may need to be improved in handling this.

EDIT: by the way, my team is still in that "delusion". It is still using global namespace and writing script files directly.... 😢

[ljharb]

That's a great discussion I have many opinions about! However, the choice of file extension here in no way answers, or obstructs answering, the question of when to build, or even if building is necessary (because of rewrite rules) - that discussion shouldn't be happening on this thread.

[unional]

Agree, as long as we universally accept that we need a build process 100% of the time. 🌷

[flying-sheep]

@flying-sheep your proposal means that I can't refactor a CJS module to be an ESM module without requiring consumers to change how they import it. I must be able to require ESM, and import CJS, transparently. "Hybrid packages" is not a complete solution to this (just like anything involving package.json), because not everything is in a package.

your point is gradual transitioning from CJS to ESM without touching importing modules?

this would only apply to projects that

1. already uses import syntax and compiles it to require syntax.
2. only uses default imports

i only know of projects that either rely on babel’s interpretation of import (where therefore import lines have to be changed anyway). i don’t even think a babel transform exists that transforms only import default syntax and complains when there’s named imports from CJS modules.