Web Modules for Node

[zenparsing]

Over the past week or so I've been experimenting with a non-transparent interop solution I'm calling Web Modules for Node. The basic idea is that instead of trying to create a single overarching module design that encompasses the strenths of both CommonJS and ES modules, we instead reframe the effort in terms of adding support for "browser" modules, similar to how node has been adding support for other traditionally browser-based features. Adding support for "web modules" may prove to be a more tractable problem.

Highlights of the MVP:

• All web modules are identified by URL.
• No file extension searching.
• No directory "index" support.
• Package.json export maps for mapping package imports to files.
• import.meta.require (or equivalent option) provides interop capabilities.
• --module/-m for loading the main file as a web module.

At the very least, it might be good to consider as a fallback in the event that various forms of transparent interop do not work out.

Looking forward to your thoughts and feedback.

Repository and prototype

[GeoffreyBooth]

@weswigham I don’t understand your “Just sayin’” example. Sure, if I control the code that I’m importing from the Web, I could make sure that it has an import map that resolves extensions; but I’m pulling from some public source like unpkg.com? If I controlled and were publishing the code I’m importing, I’d probably be loading it from my local disk.

Both browsers and Deno support code like

import BoxGeometry from 'https://unpkg.com/three@0.85.2/src/geometries/BoxGeometry.js'

I think it would be short-sighted to think that Node wouldn’t support such code someday. There is already competitive pressure to do so.

Also, that example came from https://github.com/unpkg/unpkg.com/issues/34, where a user complains about how when they try to use an import statement to pull in an ES module from unpkg.com, it fails because the above file contains code like

import { Geometry } from '../core/Geometry';

and the browser can’t resolve it. The response from unpkg.com is that the user should change the URL to https://unpkg.com/three@0.85.2/src/geometries/BoxGeometry.js?module, where the ?module is a clue for unpkg to rewrite all the URLs in the import statements to have full extensions.

So yes, the ecosystem can work around Node’s divergence, but I don’t find this a good user experience; and it’s a burden on the rest of the JavaScript ecosystem.

[weswigham]

Large import maps would also likely block render... so this isn't really a solution. Just because it "is possible" doesn't mean it would ever be used in practice

If you care about bytes shipped, you'll never use import maps at all because all bytes in an import map are extraneous compared to instead rewriting all your imports (and paths-on-disk) to the fewest bytes (ie, 1-2 characters per) with tooling. Import maps are definitely a convenience feature for devs coming from nodejs.

[weswigham]

Also why would an import map block render on it's own? Wouldn't it be async and not block until a render-blocking script that uses an import (and so depends on it) exists?

[weswigham]

but I’m pulling from some public source like unpkg.com?

You'll need to pull an import map associated with it, too (or have precalculated one yourself). Otherwise how will you map it's transitive non-relative-path dependencies, anyway? You need import maps at every step. And once you're using them, I don't see a reason not to use them fully.

[MylesBorins]

@weswigham import maps are not composable

[GeoffreyBooth]

Import maps can provide a workaround, yes; but so can configuring one's server to serve file.js for file. If I control the server and can configure it, or I can put an import map there, or just transpile the js for browsers, this isn't an issue. But users want to use code from URLs they don't control, like unpkg.com or jsdelivr or GitHub. If the author of the package hasn't made the extra effort to make their ESM code browser compatible, or included an import map that resolves extensions, then anyone trying to use it in a browser won't be able to. Browser compatible should be the default, not an extra build step.

[ljharb]

@mylesborins i believe they are composable in the latest spec.

[zenparsing]

Following up on some comments/questions upthread.

@Fishrock123

Node.js isn't a web browser and, in my opinion, adding something that is purely due to limitations in a not-node environment should not need to be part of node.

I hear you. I think one of the main reasons for node's success was that, going way back, the design effort was strictly focused on creating a really good server environment, instead of making compromises for the sake of compatibility with the web platform. The pendulum has swung the other way now, but I think that we should remember this design history.

On the other hand, the web has lead the way with (non-transpiler) ES module implementations and at this point I'm not sure that it will help to have effectively 3 module systems (CJS, ESM-web, ESM-node) rather than 2. (To clarify, I'm counting ESM transpiled to CJS as CJS.)

@bmeck

All web modules are identified by URL. What do you mean by URL here. URL string format for cache key / resolution, any arbitrary URL?... etc.

It's the same idea as the current --experimental-modules / ecmascript-modules semantics: URLs are used for resolution and as the module loader cache key.

What happens to files without extensions / hashbangs since they cannot use flags

For the MVP, the default loader refuses to load files that don't end in ".js" or ".mjs". This could be relaxed in the future. For instance, we could decide to have the default loader assume file targets are UTF8-encoded JS module code, similar to how the CJS loader assumes CJS scripts.

For the MVP, hashbang scripts need to be CJS.

"package.json export maps for mapping package imports to files." , all of this

Bare package specifier resolution works by first splitting the specifier into "package name" and "inner path" parts. It then crawls up node_module folders looking for a folder with the correct package name with a package.json file inside. The inner path is then matched against the "exports" key in that JSON file, according to the exports map proposal.

• Example package.json
• Possibly buggy prototype algorithm

What happens for import() from other contexts (indirect eval Script / Function / CJS / etc.)?

In the prototype linked from the repo, scripts are compiled with a HostDefinedOptions that contains a (possibly undefined) URL string. The default loader uses that as the base URL for resolving imports. If there is no URL string associated with the script, then the current working directory is used as a base.

Why use import.meta.require vs the alternatives? +discussing previous concerns about it.

We could get into the weeds on this one, but the main requirement is that this mechanism should not require any "plumbing" steps to get working. Users should not have to go to SO to remember the steps to enable CJS require.

How are formats determined/through what system can that be modified and/or virtualized? (asking because this seems to be a waterfall style attempt at a complete proposal of all features)

Obviously virtualization of loading is not a part of this MVP, but the architecture I have in mind goes like this: each V8 context is associated with an optional module loader via SetEmbedderData. The default context is associated with the default loader. New contexts do not have a loader, and there is no way to give them one in the MVP. Post-MVP, the idea is that new contexts can be created with an "import policy": basically, resolve and load hooks. If an import policy is specified, then the new context will be associated with a new loader that uses that policy.

For the default loader, there should be a way (post-MVP) to override the import policy. I don't have strong feelings on the API for this, other than an observation that most uses cases would seem to be served well by a plugin-style API.

[Fishrock123]

On the other hand, the web has lead the way with (non-transpiler) ES module implementations and at this point I'm not sure that it will help to have effectively 3 module systems (CJS, ESM-web, ESM-node) rather than 2. (To clarify, I'm counting ESM transpiled to CJS as CJS.)

If you consider "ESM-node" to "not be ESM" then you're going to have 3 regardless. :|

[zenparsing]

To clarify, when it comes to core semantics, I consider transpiled ESM to be an ESM-like front-end around the CJS module system.

[ljharb]

There’s always going to be more than one (two including CJS) module system - import.meta is allowed to differ by environment, as is specifier resolution.

The way to cohere that is to more concretely specify the language to allow for all of its environments, not to constrain some of them to the limits of others.

[evanplaice]

If the build process is relatively straightforward, supporting multiple formats isn't such a big issue. Most older libraries have been running multi-mode packages (ie IIFE/AMD/UMD/CJS) for years. Ideally, the long-term goal is to have ESM-first sources that can be built to support CJS and -- if absolutely necessary -- legacy JS (Ie ES5).

If you haven't heard, the Pika Package Manager just came out. It provides an interesting approach to solving this problem. It tree-shakes all but the top-level imports of a package and generates a single source file for use in the front-end.

Having used JSPM/System.js in the past, source maps work great for development but the overhead of requesting an entire tree of dependencies is too bad for prod; making prod bundles a necessity.

If we can get back to a mode where per-package (ie rather than split build bundle blobs) are the norm, then we'll we may see use of caching and loading libraries from centralized CDN sources (ie with local fallbacks) again.

As far as the Node side of that goes, as long as Node.js provides native ESM without any breaking quirks (ie .mjs required, alternative usage of pkg.module) then cross-platform compatibility shouldn't be an issue.

[GeoffreyBooth]

As far as the Node side of that goes, as long as Node.js provides native ESM without any breaking quirks (ie .mjs required, alternative usage of pkg.module) then cross-platform compatibility shouldn’t be an issue.

Can you please explain this part? What would be a breaking quirk that would imperil cross-platform compatibility, especially with regard to "module"?

[devsnek]

Your results will only include packages that are built with modern ESM syntax & include a defined "module" entry point in their package.json manifest.

damn shame cuz most of these are intended to be used with webpack/babel (non-spec esm) anyway :(

[GeoffreyBooth]

damn shame cuz most of these are intended to be used with webpack/babel (non-spec esm) anyway :(

Yes, this is what I was getting at with suggesting that Node ships defaults that encourage NPM packages to be browser-usable without rewriting. Obviously services like Pika and unpkg and jspm.io can rewrite JS on the fly to add missing file extensions or bundle files together and so on, but I don’t think it’s a good thing for users to be relying on third-party services for things like that. As you can see with Pika, if we leave this design space empty others will rush in to fill it, and we might not like what decisions they make.

[devsnek]

@GeoffreyBooth 🤷‍♂️ i'm fine with letting others fill this space, i think it makes sense to let people define their own subsets of node's larger feature set. i think this group has a bit of an over-representation in people concerned about browsers, not that being concerned about browsers is bad. in the end though, a lot of code for node can't run in browsers anyway, so letting people define their own experience makes the most sense to me.

[evanplaice]

I think this group has a bit of an over-representation in people concerned about browsers

Sounds like you assume it's a binary choice between browser-or-server. Many of us work in both. Many of us develop/maintain libraries that serve both ecosystems.

In addition to client/server development I'd also argue that CLI tooling is equally as prevalent. The JS tooling ecosystem is arguably the most mature of any language at this point.

For many of us, unifying JS across platforms will put an end to years of pain trying to support a broken ecosystem of of almost-but-not-quite-good-enough module formats. Besides, structuring a public API with ES imports is just, nice...

[devsnek]

@evanplaice i don't think it's a binary choice. i also don't think "the capabilities of a browser" is a good place to draw the line for node. node can go further.

Many of us develop/maintain libraries that serve both ecosystems.

this is my point, that it's already possible. you just have to maintain the hygiene of isomorphism instead of node doing it for you.

[jaydenseric]

Around 85% of the code that I write or publish is isomorphic/universal. For the remainder, I don't want to have to configure different linter configs, etc. for each environment.

Universal JS is the future. Differences between browser and server environments are hugely disruptive. For example, I've had nightmares trying to test in a Node.js environment libraries for interacting with standard fetch, FormData, File, FileList APIs. There are lots of gotchas with making browser code not crash in SSR that would not exist if Node.js valued commonality with browser APIs.

We are about to see a new generation of tools for shipping native, standard ESM with URL import specifiers for browsers and Deno. If Node.js doesn't have the capability, the community will be forced to employ a range of additional clunky tools.

Don't discount how hard it would be for new devs to learn the differences between browser and server environments if something as fundamental as imports worked differently:

• Node.js: "In a browser, you can import React from a URL like this... but if you want to SSR, because Node.js is different you need to setup Babel to transpile the …" etc. etc.
• Deno: "Import React like this". It's the same everywhere; easy.

Node.js will fragment the JS community and miss out on a lot of the packages and tools that will be published for Deno and browsers if it doesn't support standard import URLs and standard APIs like fetch.

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

[devsnek]

deno has a bundler built into it so I don't think that's a fantastic comparison for universal js.

aside from that, I've pushed for more common APIs between node and browsers, but it's a slow process and things don't get done unless someone volunteers their time.

none of this is relevent to the topic at hand though:

as long as node supports the subset of behaviour that deno/browsers/etc support, it's fine for extra stuff to exist on top. I don't see anyone complaining about deno filling the ecosystem with typescript, which browsers obviously can't run.

[evanplaice]

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

@jaydenseric Keep an eye out, this group exists to address the issues you mentioned. We just reached consensus today and -- if all goes well -- will be shipping it flagged in about 6 weeks.

[GeoffreyBooth]

Has anyone considered the obvious option of publishing a major new version of Node.js that simply supports ESM only? Go cold turkey on CJS. No need for the .mjs extension. The sooner CJS is in the rear-view mirror the better.

@jaydenseric Keep an eye out, this group exists to address the issues you mentioned. We just reached consensus today and – if all goes well – will be shipping it flagged in about 6 weeks.

To be clear, we’re not dropping CommonJS. But we are aiming to support ESM in ways that are more compatible with user workflows and browser equivalence. The latest plan is here.

[MylesBorins]

Can this be closed?