Special treatment for package.json resolution and exports?

[ctavan]

📗 API Reference Docs Problem

• Version: 14.2.0
• Platform: any
• Subsystem: esm

Location

Section of the site where the content exists

Affected URL(s):

• https://nodejs.org/dist/latest-v14.x/docs/api/esm.html

Problem description

Concise explanation of what you found to be problematic

With the introduction of pkg.exports a module only exports the paths explicitly listed in pkg.exports, any other path can no longer be required. Let's have a look at an example:

Node 12.16.3:

> require.resolve('uuid/dist/v1.js');
'/example-project/node_modules/uuid/dist/v1.js'

Node 14.2.0:

> require.resolve('uuid/dist/v1.js');
Uncaught:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/v1.js' is not defined by "exports" in /example-project/node_modules/uuid/package.json

So far, so good. The docs describe this behavior (although not super prominently):

Now only the defined subpath in "exports" can be imported by a consumer: … While other subpaths will error: …

While this meets the expectations set out by the docs I stumbled upon package.json no longer being exported:

> require.resolve('uuid/package.json');
Uncaught:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './package.json' is not defined by "exports" in /example-project/node_modules/uuid/package.json

For whatever reason I wasn't assuming the documented rules to apply to package.json itself since I considered it package metadata, not package entrypoints whose visibility a package author would be able to control.

This new behavior creates a couple of issues with tools/bundlers that rely on meta information from package.json.

• So right now, packages that do contain information to be consumed externally (e.g. by bundlers) in their package.json have to add the package.json to the pkg.exports field.
• On the other hand bundlers/etc should then handle the case where a dependency doesn't export the package.json gracefully, since it might very well be the fact that a given package doesn't need to export any bundler meta information (and otherwise almost all packages on npm that could ever be used in a react-native project would have to add package.json to their exports).
• If bundlers however handle a missing package.json exports gracefully, I see the risk that many modules that currently rely on their package.json simply being externally consumable without additional effort might suddenly behave in odd ways when the meta information from package.json is no longer readable by bundlers (and no error is thrown).

Examples where this issue already surfaced:

• https://github.com/uuidjs/uuid/issues/444
• https://github.com/sveltejs/rollup-plugin-svelte/issues/104
• https://github.com/react-native-community/cli/issues/1168
• https://github.com/ai/nanoevents/issues/44#issuecomment-602010343
• https://github.com/facebook/react-native/issues/28710
• https://github.com/locize/i18next-locize-backend/pull/326/files
• https://github.com/eemeli/make-plural/issues/15

Now the question is how to move forward with this?

1. One option would be to keep the current behavior and improve the documentation to explicitly warn about the fact, that package.json can no longer be resolved unless added to exports. EDIT: Already done in https://github.com/nodejs/node/commit/1ffd182264dcf02e010aae3dc88406c2db9efcfb / Node.js v14.3.0
2. Another option would be to consider adding an exception for package.json and always export it.

I had some discussion on slack with @ljharb and @wesleytodd but we didn't come to an ultimate conclusion yet 🤷‍♂️ .

---

• [x] I would like to work on this issue and submit a pull request.

[jkrems]

I think a solution where tools are allowed to assume that exports includes an entry for ./package.json would be less desirable. If tools can't be changed to use anything but require for loading metadata, the fix is to make sure require always returns the metadata. I don't think a situation where node considers one thing expected behavior (exports explicitly lists possible specifiers and exporting package.json is a choice) and ecosystem tools assume another behavior as expected (exports always has to include ./package.json) would help here.

If anything, making ./package.json implicit would be better because it may allow tools to skip it when distributing packages. It definitely creates a gray area where a minimal web-ready distribution now has to decide if it wants to risk excluding package.json because it will always technically be part of the exported files. Which is unfortunate but it may be a price worth paying to support existing node-focussed tools.

[wesleytodd]

Any chance of working with package managers to update the behavior of package.json generators?

This is a long process. I have been working with folks at npm on some other updates, it just requires a fair bit of work to make progress on. I don't think relying on generators across the ecosystem updating to make this work.

I don't think a situation where node considers one thing ... and ecosystem tools assume another behavior

I am not clear on your meaning here. While I agree tools and node doing something different is really undesired. But I think the goal here is to have one clear defined behavior which includes a good way to access the package metadata. Is that what you mean in your second paragraph?

The second paragraph also talks about browser compatibility, which is a different thing which might warrant more description to understand the use case.

[guybedford]

I've added the modules agenda label and included it in the list for the meeting tomorrow. This has come up late so it's at the end of the agenda so whether we get to it and how much progress is made will be time dependent.

[ctavan]

Given the 14.3.0 release yesterday the documentation part of this issue is IMHO now sufficiently honored in https://nodejs.org/dist/latest-v14.x/docs/api/esm.html#esm_package_entry_points

Since the discussion now mostly centers around how to deal with exporting package.json I have renamed the issue description accordingly.

[GeoffreyBooth]

So I'm not opposed to creating this package.json exception, but I would prefer if we didn't because of the education concerns (makes "exports" harder to learn and therefore slows adoption) and because of the potential for bugs (now there's one more code path in our loader, users might think that package.json being available when they didn't export it is a bug, etc.). And if someday some other tool reads the "exports" exported paths, perhaps to generate an import map, is it supposed to include package.json or not? This exception just makes the whole feature permanently more complicated, for what feels to me like a fairly small temporary gain of allowing a few tools to not need to upgrade.

Which makes me wonder, just how many tools are we talking about? If it's in the dozens or even the low hundreds, then I think a better solution is for those tools to just update to use a different method for getting this package metadata. I read through some of the linked issues to this one and apparently plenty of tools haven't had this issue, like Webpack and Rollup. I think it's premature for Node to jump to a solution when we don't have a good sense of how widespread the problem is. It's not the case that all package authors need to add package.json as an export—it's only that tools looking for a /package.json export need to use other means of finding it, like Webpack and Rollup already do.

[bmacnaughton]

This will break APM which, for better or worse, often has to patch non-public elements of a package.

package.json is imported for every package we monkey-patch. We need the version which very often determines what needs to be patched in the package.

One example of a non-public interface requiring patching in order to maintain context is mongodb for which it is necessary to patch each of the mongodb/lib/core/topologies/ files: server.js, replset.js, and mongos.js. If it is not possible to require and patch files that are not exported we will fail. There are many other examples I can list if it would be helpful.

[bmeck]

@bmacnaughton is there any reason to do this by placing it in the module cache/through the module loader compared with using fs to read these instead of package.json? per https://github.com/nodejs/node/issues/33460#issuecomment-630452758 . Not attempting to state pro/con here, just trying to see if a fs solution would be adequate instead of module loading. In particular modules are generally treated as singletons and share state/mutations over time and are put into a cache/persistent storage.

[ljharb]

(note that "using fs" still requires a way to reliably get the package.json path, which exports has disrupted)

[bmeck]

@ljharb yes, but the question for me is still if the file actually needs to be loaded through the module system vs just resolved.

[devsnek]

for apm we need a reference to the live exports.

[bmacnaughton]

is there any reason to do this by placing it in the module cache/through the module loader compared with using fs to read these instead of package.json it's not clear to me what you're asking (there is a lot to absorb in this subject area and i'm not as deeply steeped in it as you are), so i'll try to go through the possibilities as a way of exploring the range of solutions.

it's possible for APM to read package.json with fs, provided we know where package.json is. we never mutate package.json and it is a well defined format, so there is no need for us to acquire it via import. but as the module system has already resolved, read, and converted the text to an object, it seems more straightforward to not duplicate the code. providing APM with the results of the resolution, reading, and parsing is optimal from the APM perspective.

is it possible for APM to read any file it needs to patch with fs? yes, we can read the files but the implication here is (i think) that we patch source as opposed to the live exports. i believe that's possible but it's highly undesirable for two reasons that are top of mind. 1) any source changes, not just API changes, have the potential to break patching code and 2) multiple loaders working in this fashion would each make source changes even if the API didn't change.

here's pseudo-code (ignore errors, checks, etc.) for how our monkey-patching works now, with CJS:

function patchedRequire (name) {
  mod = realRequire.call(name);
  if (!builtinModule(name)) {
    pkg = realRequire.call(`${name}/package.json`);
  }
  if (!patched.get(mod)) {
    mod = realRequire(ourPatchingCode[name])(mod, {version: pkg.version);
  }
  if (realRequire.cache[path]) {
    realRequire.cache[path].exports = mod;
  }
  return mod;
}

// the patching code generally wraps various functions/classes/constructors with
// code that maintains async context. certain modules, like http/https, are modified
// to create context for instrumentation. in order to do this with many packages,
// files other than published API entry points must be patched. 

function ourPatchingCodeCassandra (mod, info) {
  const version = info.version;
  ...
  if (semver.satisfies(version, '>=3.3.0')) {
    RequestExecution = requirePatch.relativeRequire('lib/request-execution.js');
    PrepareHandler = requirePatch.relativeRequire('lib/prepare-handler.js');
    patchSendOnConnection(RequestExecution.prototype, sendOnConnection, connection);
    patchPrepareHandler(PrepareHandler);
  } else {
    ...
  }

I've played a bit with bmeck's node-apm-loader-example (and updated it to work with the current API for the resolve() hook and the defaultResolver(). As it is, it's possible to use it in a similar fashion for the top-level packages. But APM needs access to files that are not exported. And devsnek is right - APM needs access to the exports, not just the source file, in order to instrument modules reliably. An additional presumption is that whatever the resolve() hook returns is what goes into the cache; if not, then some means of replacing the cache is necessary.

[jkrems]

I think one interesting thing of note here is: APMs currently load package.json via require while node itself doesn't. Which means that APMs cause all the package.json metadata to be parsed twice and retained in memory twice. I personally would love to see an official "get metadata" API that returns effective package metadata for a given specifier/context pair, as seen by node. But it would require that node retains the version field (which it currently doesn't).

[ljharb]

@jkrems presumably it would require node to retain 100% of the package.json data, since any of it might be needed by tooling?

[bmacnaughton]

@ljharb - our APM instrumentation only uses the version at this point; it might need to know about the exports section for esm.

[bmeck]

I think an important thing that I missed was the chaining. Currently package.json meta-data has no hook for loading but could be replaced for importing. This seems like we are missing some hook so I'd agree with @jkrems

[jkrems]

@ljharb I think more general tools that need the entire package.json wouldn't mind using a userland solution that does something similar but with more configuration options and a more verbose cache. I'm not sure the same answer has to apply fully to APMs and bundlers/linters/etc..

[wesleytodd]

I think more general tools that need the entire package.json wouldn't mind using a userland solution that does something similar but with more configuration options and a more verbose cache.

I am writing such a tool (and have written others in the past), and I don't want any of these. I want simply what require does pre-exports. It works really well, is simple, and is well understood/documented. Any userland implementation will probably be less satisfactory in all three of these measures. Also, it seems like this is a regression in developer experience if what worked for years now requires new userland modules to get the same behavior.

[bmacnaughton]

But it would require that node retains the version field (which it currently doesn't).

for the purposes of APM it seems that keeping version, type, and exports would be all that's needed. I think being able to inquire the version of package that is loaded would be a broader benefit. I understand APM is not a core use case but it would be great if this could be worked in.

[guybedford]

This was discussed in today's meeting, and we were primarily looking at https://github.com/nodejs/node/issues/49445 as a solution to this problem more generally.

Would a feature like that resolve the primary concerns here?

[ctavan]

From my perspective as the original reporter of this issue nodejs/node#49445 would provide a clean way of resolving the package rout. From inspecting the tools that I found to be affected it should be possible to use this new way of resolving in all of these modules.

However it would still require every tool that currently uses require.resolve() to adopt this new API. So while we would not need all package authors to include package.json into their exports we would still require all affected tool authors to fix their tools, at least we have O(#tools) << O(#npm packages).

So from my perspective to answer whether that's an acceptable solution or not really boils down to whether we want to put this burden onto all tool authors or not (since I'm not an author of any such tool, I don't have a strong opinion here).

[wheresrhys]

I've read a fair bit of the discussion after being hit by this behaviour today. Not sure which solution I'd favour, but thought I'd feedback that the thing I find disappointing is that this change was released in a minor version, despite causing behaviour that breaks many projects.

If the impact was not anticipated, could your integration tests be more representative of the ecosystem?

If the impact was anticipated, but considered not to be a bug as userland should not be relying on this behaviour anyway, could it have been communicated more effectively? All the publicity around ESM arriving in nodejs that I've seen focuses on what it adds, with not even the official announcement mentioning that there are other effects too.

Considering v12 is LTS, it's really surprising that a deliberate change introduced in a minor bump of the runtime breaks my code.

[ljharb]

@wheresrhys to be fair, the addition of “exports” is almost always semver-major, and it’s the packages that add it, not node, that would break things.

[wheresrhys]

Running a package on node 12.6.3 works, running it on 12.7.0 breaks. The only change is in nodejs.

While I agree it is nuanced as to where exactly in the interplay between package, runtime and consumer the responsibility lies, it's clear (because it's the only thing that changed) that at least some falls on nodejs' decision to release this change outside a major.

It cost me a couple of hours of headscratching to figure out what had gone wrong. Multiply that across the nodejs community and that's a lot of disruption due - at least in part - to this decision. I just hope lessons are learned in how you push out changes in non major releases in future as I think, in this instance, (with the benefit of hindsight :joy:) it was forseeable that it'd break things.

[ljharb]

@wheresrhys which package? I assume it’s one that added “exports”?

[richardlau]

FWIW there was concern raised before ESM was unflagged in Node.js 12 and it was discussed in the Release WG and TSC (both meetings were recorded) prior to going ahead. It was acknowledged by people involved with the modules team that there was risk that the change could result in an observable change in behaviour.

[wheresrhys]

@richardlau Thanks for sharing. I suppose then I'd ask if the WG have clear criteria for assessing risks and deciding what to do about them? There are some comments in the thread above (I'm not sure if they are all by WG members) which suggest the scale of the risk may have been underestimated. Also the degree to which nodejs is responsible for mistakes/misconceptions by maintainers of the other parts of the ecosystem (compare this with the "don't break the web" ethos of TC39, W3C, WHATWG)

[richardlau]

@richardlau Thanks for sharing. I suppose then I'd ask if the WG have clear criteria for assessing risks and deciding what to do about them? There are some comments in the thread above (I'm not sure if they are all by WG members) which suggest the scale of the risk may have been underestimated. Also the degree to which nodejs is responsible for mistakes/misconceptions by maintainers of the other parts of the ecosystem (compare this with the "don't break the web" ethos of TC39, W3C, WHATWG)

As far as the Release WG goes (can't speak for the TSC), the answer is no. Perhaps the risk was underestimated, but the resolution of the release meeting was that the unflagging was allowed on the proviso that if it did turn out to be disruptive to the ecosystem it could be reflagged. I'll stick this on the release agenda for next week's meeting for discussion -- @MylesBorins since you're in both the modules team and Release WG perhaps you can bring a recommendation as to what we do for LTS (12.x) in light of this issue?

[MylesBorins]

My personal opinion is that there is no reason to reflag.

Yes this has been somewhat disruptive I think reflagging modules at this point would be substantially more disruptive

[MylesBorins]

Also taking a quick look at Node-fetch. They introduced package.exports in 3.0.0 as a breaking change. While the does mean that the package will work on 12.16 and not on 12.17 the package itself explicitly broke that interface in a semver major change. It can undo the breakage by explicitly exporting the package.json, which @wheresrhys has opened a pull-request to fix.

This is entirely inline with what we expected could be broken, and exactly the remediation process we figured could be followed. Yes it is breaking, but it is broken by the interface of the package in a way that can be fixed without any changes in Node.js core. This is, imho, no different than a package begining to use any other Semver-Minor addition to node core, it will break older versions of the same minor that do not have that feature.

[wheresrhys]

@MylesBorins I agree with most of what you say. I'm not advocating for reflagging - I agree it'd be immensely disruptive - but I think there's still room for reflection as to whether nodejs should tread more carefully in future.

This is, imho, no different than a package begining to use any other Semver-Minor addition to node core, it will break older versions of the same minor that do not have that feature.

This is conceptually quite a complex statement. How responsible nodejs is for mistakes in understanding and implementation by package publishers is debatable (somewhere between 'not at all' and 'a little' I guess), but, having identified there was a risk, better communication about that would've helped.

[eemeli]

As a package maintainer who did start to use "exports", what was not obvious to me was the existence of tools such as Facebook's Metro bundler that will fail if they can't resolve pkg/package.json. It would have been useful (and possibly still is) for that to be better documented in the ESM docs.

In other words, even if I cannot conceive of a reason to need to have package.json exported, at least for any code that could end up in a front-end application, there absolutely are tools that depend on it.

[jkrems]

[...] the existence of tools such as Facebook's Metro bundler that will fail if they can't resolve pkg/package.json.

I tried to find an issue on their side but couldn't find one. Was this reported to metro already?

[ctavan]

I tried to find an issue on their side but couldn't find one. Was this reported to metro already?

@jkrems you can check https://github.com/eemeli/make-plural/issues/15 for the relevant references.

[jkrems]

Looks like this is the issue on the metro side (linked from make-plural): https://github.com/facebook/metro/issues/535

[MylesBorins]

As this is still on the module agenda and there is no action item for the release team I'm dropping the release agenda tag

[jedwards1211]

I guess scanning node_modules and adding missing ./package.json entries to exports is easy enough to automate... :trollface:

[MylesBorins]

Dropping the label. @ljharb is going to work on this

[VincentBailly]

I just published a few lines of code that allow to resolve the location of a package with less disk IO than require.resolve and that is not broken by exports. Feel free to use, copy, fork, improve or ignore it :) https://www.npmjs.com/package/package-resolver

[ljharb]

@VincentBailly i believe that will fail if the package has no “main”/“.” (packages might not have any exports at all), and doesn’t handle packages in node_modules directories above the “from” (like, inside a monorepo, for example). Also, have you tried scoped packages?

[VincentBailly]

@ljharb have you looked at the code?

• The few lines of code don't even read the content of the package.json, the only thing that matters is that there is a folder with a given name under the node_modules folder.
• It works with monorepos, the algorithm is recursive and keep going up the file system until it finds the package.
• It works for scoped packages (eg. @foo/bar)

[ljharb]

ah, i see that path.join would handle that. Thanks for clarifying; i'd misread it. That's definitely not exactly the same as "what require.resolve would do", but it's a reasonable approximation for common cases.

[MylesBorins]

This is great @VincentBailly ... I wonder if we could add it to module as a utility method (similar to what was proposed above)

[ljharb]

For node core to support it, i'd hope/think we need it to be bulletproof - ie, to actually use the resolution algorithm.

[MylesBorins]

@ljharb I think opening a PR with this as a start is a great place to begin, then we can iterate / add tests to get to a place where we can land in core. Just trying to ride the momentum :D

[tomkretzschmar]

Is there any update for this issue?

[lostpebble]

I'm confused about why this is a concern; if you put secrets in a place on the filesystem that the node user can access, your secrets are already exposed. exports is not a security feature, as we discussed many times during its development.

To pick up from what you said earlier @ljharb , exactly this. It should not be a concern as module consumers in Node.js- we have access to the file system and can, through our own hacks (like @VincentBailly provided above), find whatever file we would like. There can be no expectation of privacy where we have control over all internal server processes.

This whole issue basically comes down to us wanting to make use of Node.js's module resolving algorithm for file system lookups- in situations where we do have this ability and privacy surrounding this action is not a concern. This isn't about actually consuming the module as defined in exports. If require.resolve needs to have parity with require(), then we simply need a different method altogether that can achieve this. Something like require.resolveModuleBasePath("module-name") or even something not namespaced to require at all.

The fact that we've lost the ability to do a simple require.resolve("module-name/some-module-file.json") is very detrimental to many development processes- thankfully, it can and will be worked around.. but for developer safety and not shooting ourselves in the foot by selecting the wrong module path, it would be much appreciated to have the native functionality available. Surely this path resolving algorithm is all already in there for the current functionality, it just needs to be exposed under something else.

I don't know if making an exception for just a single file (package.json) is the way to go- just give us the native "bulletproof" module resolving algorithm for us to use as we please (as you mentioned above). Then we can just do:

const packageFile = path.join(require.resolveModuleBasePath("module-name"), "package.json");

---

EDIT:

For anyone who needs a "quick and dirty" way to use the actual Node.js module resolve algorithm. I've written a small function which seems to be doing the job well (I have a Webpack plugin that needs to resolve hundreds of module paths during compilation, and so far seems to be working without a hitch same as the functionality was before the whole .exports limitation stuff):

function resolveModuleBasePath(moduleName, options = undefined) {
  const moduleMainFilePath = require.resolve(moduleName, options);

  const moduleNameParts = moduleName.split("/");

  let searchForPathSection;

  if (moduleName.startsWith("@") && moduleNameParts.length > 1) {
    const [org, mod] = moduleNameParts;
    searchForPathSection = `node_modules${path.sep}${org}${path.sep}${mod}`;
  } else {
    const [mod] = moduleNameParts;
    searchForPathSection = `node_modules${path.sep}${mod}`;
  }

  const lastIndex = moduleMainFilePath.lastIndexOf(searchForPathSection);

  if (lastIndex === -1) {
    throw new Error(`Couldn't resolve the base path of "${moduleName}". Searched inside the resolved main file path "${moduleMainFilePath}" using "${searchForPathSection}"`);
  }

  return moduleMainFilePath.slice(0, lastIndex + searchForPathSection.length);
}

[byCedric]

I'd love to add more context from the React Native side of view. With RN, and probably other cross-platforms, we ship native code and configuration files with Node modules. This can be anything from Android's Gradle/Java files to iOS's Podfiles and Obj-C code. These files are linked from the node_modules folder into the native side. This process requires fetching certain meta-information about the package, such as is the installed module version compatible and installed location of the module (e.g. for monorepos).

By blocking access to package.json through exports, we lose exactly what @lostpebble talked about, the ability to find the meta-information we need to connect the module to the native side properly. Adding package.json to the exports does seem to give us some of the abilities back. However, that approach does assume certain things that might be better behind an internal API. I'd love to help think of specific APIs that we could use instead of adding special treatment for the package.json (if necessary).

Also to comment on:

If you don't like change, don't adopt exports.

We are not in control of what other library authors create. exports has a lot of good things, but as many others have said here, it's kind of breaking existing functionality without an alternative.

For context:

• @react-native-community/cli issue about the 'Package subpath './package.json' is not defined by "exports"' error
• expo-cli issue about Firebase v9 (with exports) and detecting the installed version

[lostpebble]

Quite honestly this is a pretty major regression in Node.js as far as dev tools go and it's breaking a lot of things. I hope the team moves fast to restore this functionality natively.

[GeoffreyBooth]

I think at this point it’s clear that there won’t be a special exception to "exports" for package.json; to do so now would be a semver-major change, I’d think.

https://github.com/nodejs/loaders/issues/26 is on the Loaders roadmap, and should address the original use case, of trying to find the package.json for a package. Basically, we would create a utility method to return the relevant package.json file(s) for a given specifier.

I think I’m going to close this as the specific request here (no. 2 above, “consider adding an exception for package.json and always export it,” not the docs improvement which was already done) seems unlikely to be implemented, but others should feel free to reopen if they disagree.

[dipendra-sharma]

One more.

https://github.com/react-native-community/cli/issues/1650