Pluggable Loaders to support multiple use cases

[MylesBorins]

As #81 is getting into implementation details of the loader I thought it might be a good idea to start discussing an idea I have been kicking around for a robust solution. "Pluggable Loaders". This is heavily inspired by https://github.com/nodejs/node/pull/18392

TLDR; (bluesky)

• Loader Plugins can be specified via an out of band mechanism (a.k.a. package.json or command line argument).
• Loader Plugins can included within a project as an es module or installed via a package manager
• Loader Plugins can specify the behavior of the loader including
  • behavior for specific extensions or mimetypes
  • behavior for lack of extensions (e.g. path searching)
• Loader Plugins can cascade
  • plugins can override default behavior

Example loaders include

• std/esm
• TypeScript
• wasm
• Babel

Questions to be answered

• Is this approach of interest?
• Should plugins be specified per application or per package
• What should node ship as default loaders?
  • Proposal
  • Package name map as default loader
  • "Legacy" loader shipped in core as well

Thoughts?

[devsnek]

this is pretty much possible with the current system's hooks. using a combo of resolve and dynamic instantiate you can do everything from hot reloading to babel-style cjs named exports.

[GeoffreyBooth]

Like I wrote in #70, if developers need to still use Babel or esm or the like in order to achieve a common use case, then many of them will just keep using that tool as they do now and have it transpile everything, and it won’t matter that Node has added native module support. So asking developers to configure such a tool as a direct dependency of their apps doesn’t seem like much of a gain to me; it’s essentially where we are now.

I can see the benefit of maybe the package to be loaded having a flag in its package.json saying “use std/esm to load me” as its way of being compatible with both ESM and CommonJS, and therefore the app developer doesn’t need to install/load esm or Babel as a direct app dependency (as opposed to a dependency of that package). But that only works as long as all of an app’s dependencies have such a flag, and it will be many years before that happens unless we intervene somehow. Perhaps the flag could be added automatically for packages that lack it, but then we’re anointing a pseudo-official package loader which essentially needs to be supported as part of Node. Which might be okay! NPM is distributed with Node and more or less supported as part of Node core, so if esm say becomes the “official” loader dependency for CommonJS modules that don’t specify an alternative, that might be a workable solution.

[bmeck]

@GeoffreyBooth I'm of the opinion that standardizing on ESM is of great value as it has the potential of sharing code without a build step between a variety of environments (not even just Node/ Browsers). If people are wishing to perform transformations that are not supported in all environments it makes some sense to me that it requires some configuration. I think continuing as we are with build tooling and required configuration should be seen as a red flag for your code working in all environments since some things like web browsers are not seeking to add such hooks at this time.

I firmly believe getting unification of hosting environments and developer ecosystems is of higher value than anything else that ESM can provide. We should seek to provide solutions to use cases, but some use cases are served in simple ways like how importing bindings by name may not be necessary if you can read properties of an object similar to how the default based import design works.

We can solve use cases in multiple ways, and that leads to the potential of multiple loaders which are tailored to specific use cases or environments in specific ways. I don't think shipping a specific loader is the best way to unify developers and actually encourages using tooling to continue to be relied on to assist in managing differences between environments.

[GeoffreyBooth]

@bmeck The issue is, ESM doesn’t offer many benefits for Node users over CommonJS. There’s no reason the many thousands of Node developers will rush to use it. Some will, sure, but many others won’t see the need, just as many packages on NPM are still unapologetically using require statements and never switched to import with transpilation. We have 600,000 CommonJS modules on NPM, that will need to still support CommonJS environments for several years. For all practical purposes, every Node developer will need to import CommonJS packages for several years to come; and the more that using ESM is an incompatible breaking change, the less people will be able to start using it or be inclined to.

CommonJS is part of Node. It’s not just some other loader, like some userland thing that maybe Node tries not to break. If you want people to start using ESM, you need to provide them a way to use it in the world we live in now, where almost every dependency they import is a CommonJS module. And if that way is to push users off on userland solutions like Babel or esm, users will keep using those tools to transpile down to CommonJS the way they’re doing now. Adoption of true ESM will be even slower.

Look, I’m all for getting to the nirvana of an ESM-only future. But you need to provide a path for people to get there, because they’re not going to just throw out every app they’ve ever written and every library they’ve ever used to start using some new spec that has no obvious advantages for them.

[bmeck]

@GeoffreyBooth I'm not suggesting they throw out applications they have written, and they can continue to use whatever compile to JS system they already do. I'm stating that the satisfaction of use cases might not match a specific compile to JS system and custom tailoring those loaders is probably going to continue. Node providing a compatible system with other environments is a higher priority to me than matching any given compile to JS system of today since people can continue to use those as they migrate. I don't think standardizing on a compile to JS system is a good idea. We have many tools using the ESM syntax for different semantics and need to admit that adopting and/or excluding loaders will have the same effect that you are seeking to avoid. You will break some amount of application, and also are breaking compatibility with other environments. I see unity in allowing loaders and creating a safe system for usability in other environments, not in encouraging semantics that enforce a specific behavior at odds with other environments.

[GeoffreyBooth]

What’s the definition of “loader” here? Can a loader be something that converts import x from 'commonjs-lib' into const x = require('commonjs-lib') at runtime? And then can this loader be included by Node automatically when there are CommonJS dependencies in a package.json? Or if that’s a bridge too far, maybe when there’s something in package.json telling Node to load it, and NPM can put that configuration there when it knows it’s needed because of the dependencies?

Because that would be fine. That would solve the “import from CommonJS” use case.

[devsnek]

that could be something that a loader could in theory do, however it causes me great stress that node transpiling code is somehow something that people would be okay with.

[bmeck]

@GeoffreyBooth there are a ton of topics covered in that paragraph. Lets go with, it could be? it might not be? to all those questions. In general a loader is something that is given control of HostResolveImportedModule in some way. "Pluggable Loaders" as described above do something that isn't the default behavior. Having Node do something automatically would mean it is default so probably wouldn't be called a loader in the context of "Pluggable Loaders" as described at the top of this issue.

[GeoffreyBooth]

@bmeck So work with me here. Describe to me a solution that you would find acceptable. My criteria for success are:

• Code like import x from 'commonjs-lib' that people have been writing for years needs to work like it does now with Babel/esm.
• Users shouldn’t need to install userland solutions like Babel or esm that transpile an entire app into CommonJS.
• Whatever the interoperability solution is, it needs to be either part of Node proper or officially supported by Node, like how NPM is.

I’m not a fan of transpilation either, that’s just meant as a shorthand of explaining what I want the loader to achieve.

Let’s say we create a loader called commonjs-loaderthat can make import statements work for CommonJS dependencies. When a user does npm install, NPM can see that their app is in ESM mode (because of something in package.json) and also that they have CommonJS dependencies. Then NPM can throw a warning like:

This project has CommonJS dependencies. Install a loader to allow importing them:

  npm install commonjs-loader

I don’t see the point of NPM not just adding it automatically unless there’s some other loader already configured, but if that makes a difference, fine. The point is that it’s not Babel or esm—it’s not transpiling the entire app into CommonJS. It’s just enabling interoperability, while keeping Node in ESM mode (aside from the CommonJS interoperability).

It does seem odd that this wouldn’t be part of Node, though, as CommonJS is part of Node. This feels like something Node should solve or have a solution for built in, like it has core modules like fs and path.

[bmeck]

• Code like import x from 'commonjs-lib' that people have been writing for years needs to work like it does now with Babel/esm.
• Users shouldn’t need to install userland solutions like Babel or esm that transpile an entire app into CommonJS.

These seem in conflict as you need to use tools to get their behavior if it has problems being converted to ESM. I would say to keep using tooling if you need the behavior of tooling and don't want ESM.

• Whatever the interoperability solution is, it needs to be either part of Node proper or officially supported by Node, like how NPM is.

As with the first two points you are mandating the behavior of existing tooling, so keep using that tooling.

Given those 3 points the only solution is to completely adopt one of the compiler chains and doing it at runtime rather than implementing ESM. I don't think mandating the behavior of tools and then saying not to use the tools makes sense. The solution of just always using a runtime compiler similar to how Meteor does things satisfies your points but doesn't seem desirable to me.

You could change the specification to comply to some specific tooling, but I'm not going to go into that since this was about what can be done today I presume.

---

The point is that it’s not Babel or esm—it’s not transpiling the entire app into CommonJS. It’s just enabling interoperability, while keeping Node in ESM mode (aside from the CommonJS interoperability).

This is done through some level compilation/manual hooking even with loaders since it has to manipulate how code functions. I think there might be confusion on how loading code is affected by loaders. Loaders just instrument the various parts of ESM. Breaking ESM constraints requires manipulation of behaviors in some way and not using ESM and/or doing code transforms.

[GeoffreyBooth]

@bmeck Is there a solution that you would find acceptable where import x from 'commonjs-module' works in ESM code? As opposed to a user transpiling that source into CommonJS before Node ever sees it, and then Node runs the CommonJS like it does now.

[bmeck]

@GeoffreyBooth We can load the module namespace still, whatever that means. --experimental-modules works today and doesn't go outside of any specification behavior. The default behavior could expand in various ways for named imports either by altering the specification or preprocessing. Those behaviors for providing named exports are better described in https://github.com/nodejs/modules/issues/81

[benjamingr]

@bmeck @GeoffreyBooth

What if we ship a command line flag (like we do for esm) today that supports named exports of commonjs modules so users get the familiar user experience but we show a warning when it is used and ship Node with a tool that users can run on their project and converts ""cjs named imports"" to destructuring assignments?

That would let users starts with running their transpiled code natively as a start with a flag as well as give them an automatic tool to transition to more compliant ESM in the future. We can give babel and TypeScript users transforms that do this automatically too.

Named exports would work between ESM modules anyway so the UX on those isn't hurt.

[bmeck]

@benjamingr the problem is the nature of "supports named exports" with that. If we can do it, it should be on by default and stay supported. It requires code transformation and I would be against it by default since it currently breaks how the specification requires things to act by doing behaviors like late binding which are not supported by the VM, or synthetic imports. Those breakages lead me to not want the idea to ever land if it is not on standards track for support by ESM.

Shipping a tool that does this transform ahead of runtime for you seems fine to me. The tool needs to not do destructuring assignment as that loses liveness and doesn't have a clear way to determine if a module namespace being imported is CJS or ESM though. The biggest problem is setting up the transformation without knowing if the import is CJS or ESM. You have to determine the mode of your dependencies in order to transform your module. That requires all dependencies be available and is not suitable for localized isolation in a package manager if that changes over time (people move to/from one format to another). So, it probably wouldn't work at the library level, but it probably would work as an application tool.

[benjamingr]

that loses liveness and doesn't have a clear way to determine if a module namespace being imported is CJS or ESM though.

Can you elaborate on why a preprocessor tool would not know that?

[ljharb]

@benjamingr not if it's import() from a URL.

[GeoffreyBooth]

We can load the module namespace still, whatever that means.

So . . . what does that mean? 😄

It sounds like we have two options here regarding getting import x from 'commonjs-module' to work:

1. Somehow find a way to make this happen within the context of the ESM spec, however this might be, even if it means going back to the committee and requesting spec changes. This sounds like a lot of what was discussed in #81.
2. Use a loader or some other plugin to change Node’s behavior regarding CommonJS, even if that means the spec is broken, because it’s something explicitly added by the user and doesn’t mean that Node proper is violating spec.

Either or both options can be pursued. @bmeck you’re an expert on the spec, so I would encourage you to propose solutions for either option. Maybe the group doesn’t decide to pursue those solutions for one reason or another, or maybe you think they’re bad ideas, but if you were forced to come up with them, what would they be?

--experimental-modules doesn’t provide any interoperability with CommonJS, and that’s the problem I’m trying to solve.

[MylesBorins]

@GeoffreyBooth I think we need to also accept that there is another possibility... we don't support transparent interoperability, and to use common-js one must use import.meta.require

[benjamingr]

@ljharb

@benjamingr not if it's import() from a URL.

I have never considered that live bindings need to work with dynamic imports... somehow that makes them even more magical 😮

That said - wouldn't the overhead of wrapping it "in case" be negligible if it's only done for dynamic imports?

[bmeck]

Can you elaborate on why a preprocessor tool would not know that?

Given a library mylibrary that does import 'foo';, it needs foo in order to determine what format the entry point is. When preprocessing like most libraries the output would be in isolation and not have the foo dependency pinned to a specific version. That means that mylibrary doesn't know what format foo is in if you preprocess it in one source tree but run it in another (like by downloading it off a package registry into a different source tree).

Applications don't have this problem as they generally have dependencies pinned / the entire source tree when they are run. They are a great time to run tools that require pinned versions and the source tree to not change.

[bmeck]

@GeoffreyBooth

--experimental-modules doesn’t provide any interoperability with CommonJS, and that’s the problem I’m trying to solve.

What do you mean? You can import CJS with it.

// main.mjs
import dep from './dep.js';
console.log(dep.x);

// dep.js
module.exports = {x: 12345};

[benjamingr]

@bmeck

That means that mylibrary doesn't know what format foo is in if you preprocess it in one source tree but run it in another (like by downloading it off a package registry into a different source tree).

Thanks - that explains things, but couldn't this be alleviated by requiring running the tool on npm install? (or even doing so automatically once the tool is run?)

[bmeck]

@benjamingr even if you do it on install that kind of workflows don't work with times you run with symlinked dependencies, are manually updating things in your source tree, and/or don't use npm. I had similar talking about problems of doing things at install time only in https://github.com/package-community/discussions/issues/2#issuecomment-331048793 which might help explain the situation here as well.

[GeoffreyBooth]

I think we need to also accept that there is another possibility… we don’t support transparent interoperability, and to use common-js one must use import.meta.require

@MylesBorins That’s always an option, of course, but I’d prefer that be a last resort. That means users need to keep track of which of their dependencies are CommonJS and which are ESM, and refactor their code whenever a dependency switches from one format to another. When projects usually have dozens or hundreds of dependencies, that’s a burden; though I’m sure someone will write a tool to automatically convert import and require statements back and forth as necessary depending on the dependency. At the very least, I think such a tool should be built and released before Node’s module support is publicized, so that the release notes can say “use this tool to rewrite your code for you”.

But obviously it’s more user friendly if existing code works as is without needing refactoring, automatic or otherwise.

[MylesBorins]

I personally see it as the more compelling approach for a number of reasons but am waiting until we finish determining features before pushing any particular approach

On Wed, May 16, 2018, 5:31 PM Geoffrey Booth notifications@github.com wrote:

I think we need to also accept that there is another possibility… we don’t support transparent interoperability, and to use common-js one must use import.meta.require

@MylesBorins https://github.com/MylesBorins That’s always an option, of course, but I’d prefer that be a last resort. That means users need to keep track of which of their dependencies are CommonJS and which are ESM, and refactor their code whenever a dependency switches from one format to another. When projects usually have dozens or hundreds of dependencies, that’s a burden; though I’m sure someone will write a tool to automatically convert import and require statements back and forth as necessary depending on the dependency. At the very least, I think such a tool should be built and released before Node’s module support is publicized, so that the release notes can say “use this tool to rewrite your code for you”.

But obviously it’s more user friendly if existing code works as is without needing refactoring, automatic or otherwise.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389672441, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV0kNBV_PXtRP8TAuRhB-cflRoiJvks5tzJqxgaJpZM4T-i6Q .

[benjamingr]

That means users need to keep track of which of their dependencies are CommonJS and which are ESM, and refactor their code whenever a dependency switches from one format to another.

Why? import.meta.require would keep working - switching to import when dependencies switch would make the code nicer but wouldn't be a deal breaker in this case if I understand correctly.

On the other hand it would provide worse UX than the current userland solutions arguably.

[GeoffreyBooth]

If import.meta.require can import either CommonJS or ESM, then it’s the universal loader I’ve been asking for behind the import statement. If that exists, as opposed to import.meta.require working for CommonJS only, then why would anyone write import statements? Just always write import.meta.require, and your code Just Works.

I thought one of our goals was also to get people using the standardized ES2015 syntax for imports and exports. Offering import.meta.require goes against that, especially since people will be forced to use it for years (whether or not it imports only CommonJS or both).

From a user’s perspective an import.meta.require that supports either type of module feels like Node is acting obtuse. If Node could allow import.meta.require to import any type of module, obviously Node could allow the same for import but it just refuses to, because spec.

[bmeck]

@GeoffreyBooth can you explain how import does not work for CommonJS in the example I gave above.

[MylesBorins]

I'm not imaging that import.meta.require would support esm

For dynamic esm you would use import()

On Wed, May 16, 2018, 5:53 PM Bradley Meck notifications@github.com wrote:

@GeoffreyBooth https://github.com/GeoffreyBooth can you explain how import does not work for CommonJS in the example I gave above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389677808, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecVyldAowyvDVTkb1tj49A6Wa60nOWks5tzJ_AgaJpZM4T-i6Q .

[MylesBorins]

Also import.meta.require is definitely following the standard... import.meta was specifically created for host specified stuff to give similar behavior to what we currently use an iife for in common to get stuff like require currently.

On Wed, May 16, 2018, 5:55 PM Myles Borins mylesborins@google.com wrote:

I'm not imaging that import.meta.require would support esm

For dynamic esm you would use import()

On Wed, May 16, 2018, 5:53 PM Bradley Meck notifications@github.com wrote:

@GeoffreyBooth https://github.com/GeoffreyBooth can you explain how import does not work for CommonJS in the example I gave above.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389677808, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecVyldAowyvDVTkb1tj49A6Wa60nOWks5tzJ_AgaJpZM4T-i6Q .

[GeoffreyBooth]

@bmeck I was asking about importing from dependent packages, not files. Can it do this?

// const { shuffle } = require("underscore");
import { shuffle } from "underscore";

shuffle([3, 5, 7]);

Using Underscore in this example because it’s CommonJS-only.

[bmeck]

@GeoffreyBooth you can do:

import _ from 'underscore';
_.shuffle([3, 5, 7]);

and load it using --experimental-modules currently.

[GeoffreyBooth]

Okay, so you can’t do import { shuffle } (the destructured version). I think that’s what #81 was referring to. The problem is, the destructured syntax has been the de facto standard since 2015; that’s what all the examples recommend. Sure, users can refactor, but it’s pretty annoying; and it feels like it shouldn’t be necessary, since { shuffle } = require("underscore") works. It feels like Node supports destructuring except in import statements—but only when you’re importing from CommonJS.

[bmeck]

@GeoffreyBooth named imports are not destructuring so I don't use them in my example, using destructuring would change the behavior so even if you did some form of code transformation you wouldn't use destructuring, but some form of property dispatch like _.shuffle in order to get the binding value.

It feels like Node supports destructuring except in import statements—but only when you’re importing from CommonJS.

That probably shakes out from a miscommunication / syntactic similarity of named imports and destructuring when they are different mechanisms.

[GeoffreyBooth]

That probably shakes out from a miscommunication / syntactic similarity of named imports and destructuring when they are different mechanisms.

I know. But that’s a hard distinction for users to make. And we still have the issue that they’ve been encouraged to use the named imports/destructuring syntax since 2014.

[bmeck]

named imports syntax and destructing syntax are separate things, even if they both use { } around them. named imports do not have nested structures, nor defaulted values with =. destructuring does not provide live bindings, nor aliasing with as. We should not treat them as the same nor state that one uses the syntax of the other. They are separate features.

[MylesBorins]

Could we support destructing on imported common js or would that break the interpreter?

On Wed, May 16, 2018, 6:17 PM Bradley Meck notifications@github.com wrote:

named imports syntax and destructing syntax as separate things, even if they both use { } around them. named imports do not have nested structures, not defaulted values with =. destructuring does not provide live bindings, nor aliasing with as. We should not treat them as the same nor state that one uses the syntax of the other. They are separate features.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389683521, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV6tj146xaIzkoNF8GIplSWZOt09Kks5tzKVwgaJpZM4T-i6Q .

[devsnek]

@MylesBorins can you clarify what you mean by destructing on imported common js?

[bmeck]

@MylesBorins you could always do:

import _ from 'underscore';
const { shuffle } = _;
_.shuffle([3, 5, 7]);

Per changing the specification, you could expand named imports to have something like:

import {default as {shuffle}} from 'underscore';

probably? it doesn't conflict with any lookups on a glance but doesn't have well defined semantics right now without a proposal.

[ljharb]

@MylesBorins i believe it wouldn't permit the names to be statically known prior to evaluation, which is what the syntax/spec requires.

@GeoffreyBooth it might be a hard distinction for users to make, but it's a very important one for them to understand - otherwise they might wrongly think that export default { foo: bar } can be imported like import { foo } from 'path'.

[MylesBorins]

@ljharb I'm less concerned with them being statically known, more curious if there is a way we could assume that imported cjs aleays exports an object and allow it to be destructed in the import statement

My gut is this would violate spec.

On Wed, May 16, 2018, 6:31 PM Jordan Harband notifications@github.com wrote:

@MylesBorins https://github.com/MylesBorins i believe it wouldn't permit the names to be statically known prior to evaluation, which is what the syntax/spec requires.

@GeoffreyBooth https://github.com/GeoffreyBooth it might be a hard distinction for users to make, but it's a very important one for them to understand - otherwise they might wrongly think that export default { foo: bar } can be imported like import { foo } from 'path'.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389686433, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecVzxwx05VddgRQQrubfdDCMiJD07yks5tzKivgaJpZM4T-i6Q .

[ljharb]

Since the spec requires that they be verified prior to any evaluation, i believe it would (iow, "statically known" is a spec requirement)

[GeoffreyBooth]

@ljharb Let’s please not get hung up on the fact that I’ve been referring to it as destructuring syntax. I just meant the import { ... } syntax. Whether or not users understand the proper term for it, they’ve been writing import { shuffle } from 'underscore' for three years now thanks to Babel and so they think that that’s the correct syntax—and it is, but only for ESM. As a user, I would feel like this is a breaking change for me.

But @bmeck’s larger point is well taken. If you can import a CommonJS module, the interoperability problem isn’t as terrible as I was assuming. There’s still the option of having a tool automatically refactor all of a project’s import { shuffle } from 'underscore'-like lines into code like import underscore from 'underscore'; const shuffle = underscore.shuffle;. I would encourage us to make building such a tool a prerequisite before releasing any implementation that doesn’t honor the syntaxes that users have been using in the last few years, so that they have an easy upgrade path.

But obviously it would be a better user experience if that weren’t necessary.

[MylesBorins]

I'd people are satisfied with the above for transparent interop we could potentially move forward with current implementation and work with standards for future iterative improvement

I'm saying this with my 'yes and' hat on...

On Wed, May 16, 2018, 6:52 PM Geoffrey Booth notifications@github.com wrote:

@ljharb https://github.com/ljharb Let’s please not get hung up on the fact that I’ve been referring to it as destructuring syntax. I just meant the import { ... } syntax. Whether or not users understand the proper term for it, they’ve been writing import { shuffle } from 'underscore' for three years now thanks to Babel and so they think that that’s the correct syntax—and it is, but only for ESM. As a user, I would feel like this is a breaking change for me.

But @bmeck https://github.com/bmeck’s larger point is well taken. If you can import a CommonJS module, the interoperability problem isn’t as terrible as I was assuming. There’s still the option of having a tool automatically refactor all of a project’s import { shuffle } from 'underscore'-like lines into code like import underscore from 'underscore'; const shuffle = underscore.shuffle;. I would encourage us to make building such a tool a prerequisite before releasing any implementation that doesn’t honor the syntaxes that users have been using in the last few years, so that they have an easy upgrade path.

But obviously it would be a better user experience if that weren’t necessary.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389690985, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecVwpkzMoNKlesWkMNiVj8wKlPWX93ks5tzK3IgaJpZM4T-i6Q .

[benjamingr]

@GeoffreyBooth

I would encourage us to make building such a tool a prerequisite before releasing any implementation that doesn’t honor the syntaxes that users have been using in the last few years, so that they have an easy upgrade path.

@bmeck made an interesting point when I suggested that in https://github.com/nodejs/modules/issues/82#issuecomment-389654363

[GeoffreyBooth]

I didn’t say I was satisfied, just that we have a fallback option. Asking users to refactor their code feels like an admission of defeat, that we couldn’t make it work (or arrogance, that we don’t consider their time valuable). I still think it’s worth exploring if there’s a way to avoid that, even if it means a trip back to TC39.

I have lots of other issues with the current --experimental-modules implementation, starting with .mjs (of course). I think the NPM implementation is closer to what I hope the final version would be. But let’s not keep hijacking this thread.

Getting back to the point of this thread, @MylesBorins what can you accomplish in loaders? Could they be a way to bridge gaps like this?

[MylesBorins]

An expanded stab at this from #86

---

Implementation

• plugin system for loaders is introduced
  • default loader supports the resolution mechanism of the package-name-maps proposal
  • the runtime can generate the map at runtime using the filesystem, but should support the exact resolution semantics of the browser implementation
  • default loader is designed to be 100% compatible with the platform (browser + node interop)
  • loaders can be "bundled" to single meta loader
  • additional "official" loaders can be packaged in node as opt in
  • classic loader
    • "bundled" loader combining json, path crawling, native modules, etc.
    • each of these features can be individually opted in to
  • as new goals are introduced to the platform they are support by the default loader
  • wasm can start as an "official" non default loader and be merged into default loader when supported by platform
  • e.g. we can work with standards to try and support importing of JSON ()
  • loaders will define a mapping between file extensions & a custom loader
  • loaders are top-level, only supported at the application level
  • non default loaders can be specified in the package json via a "loaders" field and cascade
  • "loaders": ["@nodejs/classic"]
  • "loaders": ["@std/esm"]
  • "loaders": ["@typescript/loader", "@nodejs/loaders/native-modules", "@nodejs/loaders/json"]
  • "loaders": ["@mycompany/loader"] # same as above 😉
  • loader meta data can be bootstrapped on by transpilers and bundlers such as babel + webpack to allow for minimal configuration 💯
• import.meta.require can be used to escape hatch into cjs without requiring a custom loader
  • transpilers can handle this out of the box without referencing a custom loader to create browser compatible esm or script

transition user story

The Company has a large code base that is currently transpiling using babel. They are using import for both esm and cjs. They want to start using esm natively in their code base without transpilation

1) move to node.js esm implementation but cascade the @std/esm or @babel/loader custom loader to simplify things at first 2) transpile current app code using babel + babel loader to esm code, converting all current import 'cjs-thing' to import.meta.require without requiring direct user intervention. 3) test transpiled code and confirm it works as expected 4) remove custom loaders from package.json 5) as dependencies move from cjs -> esm update the import.meta.require statements to import

[bmeck]

I'm somewhat against one particular point but flexible on the rest given enough discussion of details probably.

default loader uses the package-name-maps proposal for resolution

This means that node loading cannot function without a preprocessing tool to create those package-name-maps as I read it. I do not see manual package-name-map synchronization as desirable or something that will be done for anything of decent sized module graph (even for small applications) without requiring a tool.

We need to find a way that does not require such a hefty configuration. Doing something like creating a module map at runtime might be fine, but I am not ok with requiring the ahead of time tooling that this seems to describe if it excludes a variety of workflows. We can just create it on boot or pass a flag if people want to use a precomputed package-name-map.

This would require also that package-name-maps move further along and have much most stringent review from the WG to ensure that other use cases are well served by them alone. I fear that instead of treating package-name-maps as a way to alleviate web concerns by performing precomputation of a resolution mapping, this idea of using them as the default is to force people to maintain their dependencies in a mapping that is not required for node's workflows nor is near the experience and convenience of being able to put a file on disk.

There also remains concerns about symlink based workflows if the idea is to match the web, see https://github.com/nodejs/modules/issues/62 which needs a tooling workflow and differing output if you want to have the keys be converted between one or the other. If the idea is to match the web, we would want to move to the cache system the web is using, which I also am against.

I have raised a few concerns about package-name-maps and the cache mismatch but have not had much traction with the members of those areas and have been told that it is "OK if they don't change their mind". Being OK with a mismatch is fine, but I don't want to mismatch and I don't think package-name-maps nor the cache mismatch have enough matching for this idea of using them and the web behaviors to be a sound idea.

[GeoffreyBooth]

I'm unclear on the user transition story. In step 2, “transpile current app code,” does that mean transpile it and keep the transpiled output as the new app code? So it's more of a conversion step, automatically refactoring the app to use import.meta.require?

And if that's the case, why would anyone bother with step 5, refactoring those lines into ES2015 import statements?

[MylesBorins]

Your interpretation was correct. It is more of a conversion. I was trying to outline that existing tools can be used for it.

Step 5 is about what happens when those dependencies change from being cjs to esm

On Thu, May 17, 2018, 11:36 AM Geoffrey Booth notifications@github.com wrote:

I'm unclear on the user transition story. In step 2, “transpile current app code,” does that mean transpile it and keep the transpiled output as the new app code? So it's more of a conversion step, automatically refactoring the app to use import.meta.require?

And if that's the case, why would anyone bother with step 5, refactoring those lines into ES2015 import statements?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/82#issuecomment-389910249, or mute the thread https://github.com/notifications/unsubscribe-auth/AAecV5xARC-i8LgtIFL5nv6Mo5UxbyN4ks5tzZjkgaJpZM4T-i6Q .