Guiding Design Principles

[MylesBorins]

The TSC released a medium post outlining the state of ESM as well as some high level design principals

• We are committed to shipping ESM
• We are committed to ESM being first class in Node.js
• We are committed to having the Node.js and Web platform as first class runtimes for modules.
• Modules installed via npm should be able to run after installation without requiring a build step.
• We are committed to supporting our current users and offering migration paths as necessary. This can be through interoperability or APIs to improve the developer experience of working between module systems.

As a new working group I think it will be important for us to make sure these principles align with our groups vision. If you are open to adopting the TSC's vision, please simply use a 👍🏽 reaction. If you would like to see changes please use this thread to offer opinions

[guybedford]

I am in full agreement with all these goals 👍🏻

In particular “modules installed via npm should be able to run after installation without a build step” is a goal I’ve been pursuing for some time through my own tooling work, and was why I created SystemJS which provides support for this through dynamic require tracing in the browser.

That said I think it could do with some clarification from the perspective of this group for the following reasons:

• The only way to load commonjs in the browser without a build step would be to (a) use dynamic require tracing in the browser which means custom source parsing in-browser, and (b) have the browser run resolution by hitting 404s on the node module lookup algorithm. (a) has a performance penalty and doesn’t cover all dynamic require cases like requiring identifiers. (b) has a performance penalty of latency which adds up surprisingly quickly as well.
• So having written a project that can do (a) and having played with alternatives to (b) seeing all the edge cases play out, I just don’t feel this is something a group like this could recommend as it feels like a fragile approach.
• This does not mean I don’t agree with the goal overall though, but I’d hope we can reframe this goal in a way that is perfectly solvable by this group.

Perhaps we can put some thought to how we can describe the browser requirements in a flexible way that covers these goals without writing implementations into a corner, including:

• Accepting that there will be many approaches to getting commonjs modules to work in the browser, and that some userland experimentation will be required to find the best approach for Es modules workflows. There can be approaches with service workers, compiling servers, precompile steps after npm install and in future browser hooks. We should embrace the flexibility here.
• Make sure that the goals of these tools are taken into account in design decisions - this is crucial. But ideally in a way that does not dictate the way such tools should be implemented.
• Setting the goal of this group to ensure we can define edge cases and interop so all those tools can agree on the behaviours and users can trust their code to work between tools. This allows specification to do what it does best in allowing the whole ecosystem to move forward with trusted conventions regardless of exact toolchain. For example service-worker-based non-build dev workflows can evolve into browser hooks dev workflows following the same conventions. This would be a huge benefit allowing and enabling creativity for these approaches in the ecosystem while not constraining the solutions.

Let me know what you all think. Perhaps we can split this into its own thread too.

On Tue, 06 Feb 2018 at 00:25, Myles Borins notifications@github.com wrote:

The TSC released a medium post https://medium.com/the-node-js-collection/the-current-state-of-implementation-and-planning-for-esmodules-a4ecb2aac07a outlining the state of ESM as well as some high level design principals

• We are committed to shipping ESM
• We are committed to ESM being first class in Node.js
• We are committed to having the Node.js and Web platform as first class runtimes for modules. * Modules installed via npm should be able to run after installation without requiring a build step.
• We are committed to supporting our current users and offering migration paths as necessary. This can be through interoperability or APIs to improve the developer experience of working between module systems.

As a new working group I think it will be important for us to make sure these principles align with our groups vision. If you are open to adopting the TSC's vision, please simply use a 👍🏽 reaction. If you would like to see changes please use this thread to offer opinions

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/11, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkiylDgQoV8x-mMG-y28Cxt_Y2hfbWLks5tR3_HgaJpZM4R6Meo .

[mcollina]

@guybedford I do not understand your last comment, and how you mention loading cjs in the browser in this situation (and how it relates to the "Guiding Design Principles"). As far as understand it, that is not part of this team activity, nor something we want to address in Node.js itself.

The only part that is related to the browser is that we would like our ESM implementation to be fully compatible, and ESM that do not access any specific Node.js things should work on both. That does not include cjs.

[ljharb]

To clarify, "modules installed via npm should be able to run after installation without a build step" to me means "no postinstall script", not "no prepublish script".

[guybedford]

@mcollina your question touches on a lot of topics... if you don't mind me avoiding the long answer for now and instead allow me to provide an example, note that supporting import.meta.require in ES modules in the browser is equivalent to supporting CJS modules in the browser. It is at these sorts of levels that these concerns touch.

@ljharb right, that would be a useful clarification, but the lines of my argument above is that we shouldn't attempt to impose such a form on workflows unless we are absolutely sure we can get consensus on such an approach for all users and all workflows.

[mcollina]

I would classify import.meta.require as a "node specific thing" for the time being.

[guybedford]

@mcollina it sounds like you're advocating removing the fourth bullet point altogether then!

[mcollina]

it sounds like you're advocating removing the fourth bullet point altogether then!

Absolutely not, in fact it's the exact contrary. If you want full compatibility with the Web Platform without transpiling and/or some runtime support, your module and its dependencies needs to be ESM and not to be using any node-specific API or modules (import.meta.require would just be such an API - if we ship it). This enable the community forces to migrate the packages to ESM in their own time. This is a much smaller goal than supporting require inside the browsers.

[guybedford]

@mcollina so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

My argument is to (1) maintain the scope of the fourth point to include loading CommonJS in the browser, but (2) as opposed to trying to provide a specific solution for this, to reduce the scope to treating this as an understood use case that things like interop and loading conventions must bear in mind, while not dictating what those exact workflow solutions must be.

I think if we choose to ignore browser workflows for CommonJS interop entirely, that would be disregarding many interesting use cases. We don't need to know the answers, we just need to appreciate the questions.

[mcollina]

so you'd like to see the scope of the fourth point reduced to esm-only workflows? That would make sense.

Not exactly. I would articulate it better something like the following:

1. CJS and ESM installed via npm should be able to run in Node.js after installation without requiring a build step - I believe this is the original meaning of the design principle as formulated by the TSC.
2. ESM installed via npm should be able to run in the Browser after installation without requiring a build step if it does not use any Node.js specific API.

Having a solution for cjs interop in the browser would be very nice. However, that would almost necessarily require some pre-processing and a runtime library. I would rather leave this problem to other groups (webpack, browserify, rollup etc..) to tackle.

[guybedford]

Ahh, I've been going by the Medium post definition of point 4 which specifically names the Web Platform here, which may explain the confusion:

We are committed to having the Node.js and Web platform as first class runtimes for modules. Modules installed via npm should be able to run after installation without requiring a build step.

@mcollina I think we're mostly in agreement actually, although while not directly tackling the browser interop problems, I hope we can keep those workflows in mind here that's all.

Browser unification will keep coming up in discussion here so I'm just trying to consider how we can frame those arguments here.... to be able to consider and appreciate the different types of browser workflows, and allow their consideration to affect decision making (eg if we find ourselves considering something that makes service worker workflows impossible, or static builds impossible, that should be a valid consideration), without trying to impose solutions or decide on exactly how we think those workflows should work (assuming that everyone will do post-compile on npm install for example, and ignoring all other possibilities).

[giltayar]

@guybedford - I believe the npm proposal touches on a solution to your problem: given that ESM definitions are static, theoretically npm install can go over all imports inside the installed modules and statically link them to their exact file, e.g. from import 'D' to import '../node_modules/D/index.mjs, thus enabling browsers to directly access the correct file. In essence, changing all bare imports to relative imports that can be understood by the browsers.

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports. Maybe, just maybe, we can solicit the help of the browser vendors and programatically supply a map from bare imports to "real paths".

So, yes, without touching the files, and without the browser vendors help, there is no complete solution, but given that npm (and yarn) are perceived as part of the solution, and given that the browsers are part of the problem :-), maybe we can ask for their help.

[guybedford]

@giltayar

Having solved the problem of the imported modules, we are still left with the problem of the imports in the application being coded (we can't modify their source code), and with dynamic imports.

This is exactly the sort of mindset I'm trying to reframe against - we cannot say we have "solved" something and then admit it is only "part of the solution". Anything which is only "part of the solution" must change over time to become a "whole solution". And so the very solution may well be something else entirely.

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration. So we shouldn't assume that it is an npm install process that does post-processing, as this approach hasn't been proven in the wild. For example linking workflows are supposed to be first class in npm, yet break down with this.

[devsnek]

if the web ever gets custom loaders npm or node could just host a little script on a cdn for a custom loader that uses node's resolve algorithm. seems pretty painless considering how annoying it is that people expect node's resolver pattern to be picked up by browser :/

[bmeck]

@devsnek you can implement custom Loaders with a proxy algorithm in a http server or service worker already (except for data: and blob: URLs, which are their own special case since they can't do relative imports). We ended up making sure these work before moving forward with designs on per package hooks.

[evanplaice]

Here's an attempt to recap and surface some of the identified issues.

1. The term modules should specifically mean ESM as they are specified in ES262

• Link to ES262-15 Scripts and Modules

2. CJS != ESM, and CJS browser interop falls outside the scope of this group

• CJS interop will be provided in some way, shape, or form in Node.js only
• CJS in the browser falls outside the scope of this group.

3. import.meta, is the spec-compliant location to put platform-specific implementations

Assuming spec-compliance is the intended long-term goal to ensure node/browser interop, import.meta should become the canonical place to define platform-specific behavior.

• in the short term, ensure backward compatibility by leaving existing built-ins unchanged
• in the short term, provide aliases to the existing built-ins as a gradual upgrade path
• in the mid-term, introduce deprecation messages for built-in usage outside of import.meta
• in the long-term provide an adapter to allow backward compatibility
• in the long-term, remove existing built-ins, import.meta becomes the default built-in

The upgrade path/transition to import.meta is an intended goal. It is a direction we are working toward, it may not actually be the final end goal. If it's unrealistic to remove the existing built-ins without breaking the NPM ecosystem they could be left indefinitely. Instead of axe-grinding over the many possible futures, lets focus on what we know and don't know.

Here's a good place to start:

As it currently stands. Import.meta is only at Stage 3. We're a bit ahead of the curve so it would be beneficial to reach out to TC39 to see what the hold up is.

Update: The import.meta is good to go, just awaiting an alternative implementation as proof before Stage 4.

4. Bare imports are not specified, this needs to be addressed

Update: Implementation specifics that address how bare imports are handled in browsers don't fit within the scope of the principles mentioned in this issue. Feel free to ignore this item.

If we follow the ESM spec as it currently stands, that means imports require direct links to the source (ex node_modules ) with the extension. I don't think I have to spell out why that would suck.

Here are the facts:

• there is no existing or proposed spec for handling bare imports
• inventing an approach may introduce interop issues in the future
• ESM support in browsers is too bleeding edge to make a solid determination at this time
• npm asset is coming soon, direct imports won't be so bad for non node_modules
• the node_modules should provide a 1:1 equivalent to the current CJS approach

If we're going to push browsers to provide a process for handling bare imports it should be:

• clearly specified and submitted as a new TC39 proposal
• may change throughout the spec process
• may take considerable time to push through to full specification

If we want to set the standard, it should be done right. Else, it has the potential to cause issues down the road if a future spec is defined that diverges from the Node-specific implementation.

5. An ESM-first approach to ESM->CJS interop may not be possible without top-level async

This topic is too complex to summarize here. See Module Interop

[giltayar]

@guybedford

Rather, we should consider that there are a class of techniques here, and we don't know where the exact solution is, and we should make sure to allow for that in consideration.

Exactly! My attempt to paint a solution may have been misguided, or not clear enough, but I was trying to argue against your following sentence:

The only way to load commonjs in the browser without a build step would be to (a) use dynamic require tracing in the browser which means custom source parsing in-browser, and (b) have the browser run resolution by hitting 404s on the node module lookup algorithm.

The funny thing is, now that I reread the sentence, is that I missed the word "commonjs" in there, and read it as "the only way to load esm in the browser without a build step...". I was trying to propose a solution for ESM.

So sorry about that, but the good thing is that it seems that we are in agreement that there can be a solution for using NodeJS ESM modules in the browser (that do not rely on NodeJS stuff) without a build step. (I think...)

Just to understand - if you don't believe an npm install rewrite is the direction to a solution, how do you believe this should work?

[bmeck]

@evanplaice

3. It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

4. Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

• npm asset is coming soon, direct imports won't be so bad for non node_modules

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

• the node_modules should provide a 1:1 equivalent to the current CJS approach

This is not entirely possible for a variety of reasons such as Errors being permanent, moving to URL based specifiers, etc.

In summation, TC39 is not the place to put resolution.

[evanplaice]

It is only waiting on shipping implementations to release channels per the TC39 process doc it needs two environments to ship it unflagged.

Copy. So, awaiting implementation proof. There's no axe-grinding over implementation specifics.

Update: Sorry about any confusion, I went back and removed any comments/statements not within the scope of this issue.

Your number 4 is actually outside the scope of TC39, you are getting confused with the web spec. TC39 does not mandate anything about specifiers or resolution.

Agreed

We should definitely push against this idea as reasons I discussed previously about splitting the package ecosystem and problems with keeping things in sync and dynamic import().

Disregard any mention of npm asset. It's outside the scope of this group.

[bmeck]

What about the case where you import a node_module that references another node_module via bare import syntax. Githubissues.

Githubissues is a development platform for aggregating issues.