[Feature Request] Package.json Root / Base Directory

[oleersoy]

https://stackoverflow.com/questions/51313753/npm-package-json-base-root-property?noredirect=1#comment89604253_51313753

[mscdex]

Posting a link as the entirety of an issue submission is not very helpful, especially if the link in question becomes invalid at some point. Can you please elaborate here instead?

[oleersoy]

Sure.

It would be nice if there was a package.json property that can be used to specify the root folder that module resolution should start from. This would be follow the same metaphor as the html base element or the baseUrl in Typescripts tsconfig.json.

For example suppose we have an install in node_modules/mypackage/src/file1. And all the files we want to import start under the src directory. If we could specify something like:

 { 
    root: ./src/
 }

We could then require('mypackage/file1');

This eliminates the need to compile typescript into a different folder and then copy the package.json file into that folder.

[patrickroberts]

If a consumer of your package decides they want to import, say, your package.json with require('mypackage/package.json'), wouldn't this additional layer of indirection make things unnecessarily difficult and confusing? I also don't think this is a good idea, as this concept is a breaking change that package managers will have to implement in coordination in order to resolve an added layer of indirection to files for build processes, tests, and other automated scripts, that are already confusing enough.

[oleersoy]

If a consumer of your package decides they want to import, say, your package.json with require('mypackage/package.json'), wouldn't this additional layer of indirection make things unnecessarily difficult and confusing?

No you would still do that exactly the same way you are doing it now.

I also don't think this is a good idea, as this concept is a breaking change that package managers will have to implement in coordination in order to resolve an added layer of indirection to files for build processes, tests, and other automated scripts, that are already confusing enough.

This is independent of package managers. Package managers are responsible for downloading a package and managing dependencies correspondingly. All this does is allow the package.json specification to tell Node that it should start looking to resolve packages from a sub directory of the package instead of the immediate package root.

For example right now if we do:

import { $ } from '@dollarsignpackage/folder/dollarsignmodule.ts`;

Node expects the package structure to be laid out like this:

node_modules/@dollarsignpackage/folder/dollarsignmodule.ts.

So developers usually create a dist folder to create this exact structure, which includes copying over the package.json file, etc. If Node had the root or base property option we could just set that and node would resolve from that directory instead.

So for example if the folder/subfolder folders are located in a dist folder we could just set root: dist, and then node would automatically resolve from dist/folder/dollarsignmodule.ts. It should be a fairly trivial change to the module resolver. Browsers already do this with the base element.

[Trott]

It should be a fairly trivial change to the module resolver.

If it is truly a trivial change, it might be easier to reason about this and discuss it if it's a pull request. A test using stub modules you create in the test/fixtures directory might uncover unforeseen issues or lay to rest incorrect ideas about problems this might cause.

For reasons that I hope are easy to understand, a lot of core devs are exceedingly reluctant to see anything change in the module resolver unless absolutely necessary, so please expect lots of questions.

I know you were just throwing out a possible syntax and not the required syntax, but root might be a problematic name if there are any significant packages in the ecosystem that are already using it for something. Just a note of something to consider; not saying it's a reason we can never do anything like this.

[ljharb]

cc @nodejs/modules

[ljharb]

Unless CJS has a way to designate "not the root of the package" as the root dir for resolution, I don't think it would be appropriate for ESM (or WASM, or any other future module type) to have that capability.

[richardlau]

Unless CJS has a way to designate "not the root of the package" as the root dir for resolution, I don't think it would be appropriate for ESM (or WASM, or any other future module type) to have that capability.

Node.js has no concept of packages (at least in CJS)(note that a package is not the same as a module). Currently the only field in package.json that Node.js even looks at is main -- I don't think it makes sense to extend this especially if it doesn't suit ESM.

[ljharb]

@richardlau right - i'm suggesting that node could look at an additional field to set the directory that all requires of the package resolve relative to - and if that feature existed for CJS, it would also hopefully work for ESM (and it would make sense for both).

[oleersoy]

Another idea would be to allow main to be a directory, or allow the file that main points to to point to a directory. That way we don't really change anything WRT to how any current node package works, but it still satisfies the use case.

[devsnek]

i think this would be better suited in ESM as a resolve hook.

what if cjs could resolve the current package it was in? e.g. your package is named 'X' so you do require('X/y/z');

[oleersoy]

@richardlau right - i'm suggesting that node could look at an additional field to set the directory that all requires of the package resolve relative to - and if that feature existed for CJS, it would also hopefully work for ESM (and it would make sense for both).

Yes it should work for anything (CSS, json, xml, ESM, ...). For example suppose we had a project with all sorts of resources that needed to be compiled and it had source with main and test directories setup like this:

src/main/css
src/main/xml
src/main/json
src/main/svg

src/test/json
src/test/svg
src/test/typescript

And all of these were compiled into a target folder like this:

target/main/css
target/main/js
target/main/xml
target/main/json
target/main/svg

target/test/json
target/test/svg
target/test/typescript

And now we wish to load package resources that have been distributed to NPM. So lets call the package @resources and set the main attribute to target/main.

...
"main": "target/main/"
...

This is the location that all the resources have been compiled to.

We do npm i -S @resources. Then load any resource from target/main with a statement like require('@resources/xml/customers.xml');

Node would then resolve the path @resources/xml/customers.xml into node_modules/@resources/target/main/xml/customers.xml.

[ljharb]

The more common case (not xml, svg, or css) would be "i have all my code in src/ and i transpile it using babel to lib/, but now all my consumers have to deep-require foo/lib/blah instead of just foo/blah".

This feature would allow any package using babel to support cleaner deep requires.

[oleersoy]

I think that the total impact would make economic sense as well as far as developer efficiency goes. For example projects like RxJS remap all compiled resources into a directory structure that is different from what we see on Github.

Personally when I need to see the source, I like to open the node_modules folder in VSCode and add logging statements etc. If I could recompile the code right there and have it it work instantly that would be a big performance boost. I no longer have to got back to the original source and recompile and reinstall and what since what I'm looking at in node_modules is essentially a mirror of the github repository, I no longer have to do a bunch of mental remapping gymnastics ... this extends to every single project that is currently remapping resources in order to be able to publish them.

[ljharb]

@oleersoy for the record, you can already do that, unless the package has gone out of its way to exclude their raw source from the npm package.

[oleersoy]

@ljharb True - In some cases. I think it's not so much that they are going out of their way to exclude sources, it's just that it makes sense to just publish compiled output since since that is what the clients will be requiring. For example for Typescript we have to compile the *.ts files into *.js and *.d.ts correspondingly for the type definitions. Once that's done it may seem pointless (And almost impossible) to publish the corresponding source, so we just package what has been compiled.

For example here is the project I'm working on that made me think that the feature would be useful:

https://github.com/fireflysemantics/validator

The distribution process works like this (Roughly): 1) Compiles *.ts files to target/src 2) Create the dist directory 3) Copy package.json to dist 4) Copy target/src/**/* to dist 5) npm publish

So now there is a mismatch between what is published and what is in github and the symmetry is difficult to correct, if not impossible.

[patrickroberts]

If a consumer of your package decides they want to import, say, your package.json with require('mypackage/package.json'), wouldn't this additional layer of indirection make things unnecessarily difficult and confusing?

No you would still do that exactly the same way you are doing it now.

That doesn't make any sense. How is require('mypackage/package.json') supposed to access the file in the directory ./node_modules/mypackage/package.json if node is internally using the value of root: './src' in that package.json file to resolve require.resolve('mypackage/mypackage.json') to ${process.cwd()}/node_modules/mypackage/src/package.json (or however that works, you get what I'm saying)? Either it accesses mypackage/src/package.json or mypackage/package.json, and if you're saying it's the latter, that's really inconsistent behavior.

[oleersoy]

How is require('mypackage/package.json') supposed to resolve?

I see your point. If the client is attempting to access resources that are essentially meta resources like package.json and not direct resources like packaged code that is in a base directory then require breaks ....

So it seems the only way to elegantly support it would be to add something like import(mypackage/corepackgeresource) which would resolve core package code only and respect the base or main property directive and allow require to continue to work as it works now.

[mcollina]

I'm somewhat reluctant to introduce this change. It would create packages that work only from a specific version of Node.js forward, and complicate maintainance for modules authors.

[oleersoy]

@mcollina after properly understanding @patrickroberts objection I agree that the semantics of require() should stay the way they are now.

However it would still be nice if Node had a way to do this. So the goals would be: 1) Keep require() the same. Full backward compatibility and same semantics moving forward 2) Give package authors the ability to publish the entire github repository to NPM while also making the code executable (Please see other comments to understand why this is not as simple as we might think).

This way we could:

npm i -S  `someproject`;
cd node_modules/someproject
code . //make a bunch edits
npm run dist //recompile the code 
cd ../.. //Go back to the client of the `someproject` dependency
npm run test // Run the tests again

That way we can more effectively debug NPM modules like RXJS or other Typescript / Any script projects that must create dist directories in order to be able to publish the ES5 source to NPM.

So we could just set a base directory in package.json and if the client chooses to do so they could use a Node provided import function to import Core package resources ... in other the source code that the package was built to provide, and not the meta data like package.json. For that we would still use require.

On of the benefits of this is that if someone is currently using babel to compile from src to lib and fast forward 3 years when those features are now 100% supported by all node distributions in production, the library maintainer can just switch the base attribute from lib to src and the raw source could be imported by Node clients that use the new import statement.

Related Material

Currently struggling with Typescript related scenario here: https://stackoverflow.com/questions/51362992/should-typescript-compile-tsconfig-path-aliases-to-relative-imports

[bmeck]

I think this touches on a generic need for intercepting incoming/outgoing requests in various ways of loading resources. I'm hesitant to do anything right now while we talk about loader hooks as I feel this feature may have cross cutting concerns with things like making files unavailable outside of the package for encapsulation purposes. Overall, I'm not sure we need this feature to be a field in package.json but feel that generic well constrained and designed hooks should work however they appear. I am personally biased towards using loader hooks.

Right now a package can point to other locations with node_modules/SPECIFIER, however there is no abstraction for a package to redirect all request inside its boundaries both incoming and outgoing.

I do think you can fully do this as you publish however at least for this specific case (things like encapsulation for private files cannot be done right now). You can always just ship your lib/ or src/ directory at the root instead of keeping them inside of directory when publishing.

[ljharb]

@mcollina that’s also true of every v8 upgrade, and every addition to core modules - a new module, or any new API or option.

[mcollina]

@ljharb None of those things introduce a new package format. We are shipping esm because that's part of the language, and that would be highly disruptive already. Adding one more package format would only complicate things in the long run. The amount of disruption introduced by this is far greater than adding a new module or API to me.

[oleersoy]

You can always just ship your lib/ or src/ directory at the root instead of keeping them inside of directory when publishing.

In my experience that's what most NPM packages that are compiled do, however this creates a dynamic where if we want to further investigate parts of a package, we need to clone the git repository, compile the package, and install the package locally from the distribution directory (lib or dist ... etc) that the package creates.

So in terms of workflow and eco system efficiency this is more costly because it: 1) Makes the life of people debugging packages harder 2) Has a more expensive setup proposition for compiled libraries

[oleersoy]

@mcollina this could be as simple as providing a new resolver function like import() that specifically targets resources that are in a designated directory.

So new libraries that load dependencies that are compiled can use import() instead of require() and that's the end of it.

I suspect that as we move to `import { foo } from 'boo'; like syntax this will naturally replace require() in 99% of all development scenarios (So I'm assuming ES6 / Typescript ) like scenarios occupy 99% percent of the primary development space for programmers using Node.

[bmeck]

Makes the life of people debugging packages harder

How so? If they compile and change things, they could always use source maps

Has a more expensive setup proposition for compiled libraries

I don't understand this comment, can you expand on what is more expensive. Writing a new package, maintaining an existing package, time to load at runtime, etc.

[oleersoy]

How so? If they compile and change things, they could always use source maps

Source maps point us from some compiled output back to the original source location. But if Typescript developers are packaging their compiled output only, then the utilization of the source map is limited IIUC.

I don't understand this comment, can you expand on what is more expensive. Writing a new package, maintaining an existing package, time to load at runtime, etc.

Sure - take this library for example. You'll see there is a createdist.js script in the base of the library. That is what creates the dist directory, compiles the typescript to that directory, and copies the package.json file. Once that's done I CD into the dist directory and run NPM publish.

So that was about 15 minutes of extra work that would not have been needed if Node supported a base attribute on package.json, not to mention the time it took to realize that this is the process that is needed.

Now since this is done, there is no clean way for me to include the entire github repository in the package. So I essentially have to publish the dist directory contents only. That is only *.d.ts files, and the corresponding commonjs ES5 *.js files. The original source is no longer included.

So if someone wants to debug something using the original source they have to clone the repository, add logging statements, update tests, etc. and then recompile and reinstall the module in the client that is using it.

[bmeck]

Source maps point us from some compiled output back to the original source location. But if Typescript developers are packaging their compiled output only, then the utilization of the source map is limited IIUC.

They can contain both source location and content by using "sourcesContent". What is missing?

Sure - take this library for example. You'll see there is a createdist.js script in the base of the library. That is what creates the dist directory, compiles the typescript to that directory, and copies the package.json file. Once that's done I CD into the dist directory and run NPM publish.

I might recommend putting the package.json used once published into the dist directory. It would reduce overall complexity for publishing.

So that was about 15 minutes of extra work that would not have been needed if Node supported a base attribute on package.json, not to mention the time it took to realize that this is the process that is needed.

Is this burden 15 minutes everytime you publish? Or could it be a project bootstrap like so many other workflows use. I would think this is minimal since all the steps described above could be automated.

Now since this is done, there is no clean way for me to include the entire github repository in the package. So I essentially have to publish the dist directory contents only. That is only .d.ts files, and the corresponding commonjs ES5 .js files. The original source is no longer included.

This seems odd to me, why do you want the github repository for an installed package instead of the interface that the package seeks to ship (via files)?

So if someone wants to debug something using the original source they have to clone the repository, add logging statements, update tests, etc. and then recompile and reinstall the module in the client that is using it.

How are they doing this? I ask because in general I would not support editing your node_modules as a strong workflow that we should prioritize. In fact, it is something I would personally discourage because it means that things in node_modules are more fragile due to ecosystem usage if we encourage that.

[bmeck]

I'm not saying the feature isn't valuable to have, just that it needs to go through more rigorous design phases and seeing how it can already be achieved today and if implemented what it would affect in the future.

[SMotaal]

One pain point I come across frequently with --experimental-modules is to determined the location of the package.json (ie the root) from within the module. Require had various ways to do this, but I have not found a way to do this without actually traversing paths from import.meta.url which seems like a waste of resources.

I raise this because I think it is related, if a module can inquire about it's root, it would be possible to further allow telling a module of any root. Although other than this issue, I did not come across a reason to need to coerce a resolution root.

[SMotaal]

@oleersoy if you have control on a package and want to force require(…) to resolve from a particular path inside the package you can consider adding a file in the location (ie "~/require.js") with module.exports = require; then in other modules you can simply add require = require(path_to_require_dot_js_file) which is extremely not recommended unless you are very careful in how you actually implement this pattern.

[ljharb]

Any mechanism for a module within a package to locate the package root would need to be present in CJS as well as ESM. Currently, you'd recursively walk upwards until you found a package.json.

[SMotaal]

@ljharb I am totally for the CJS == ESM aspect, I just figure that with require.resolve, ESM is at a disadvantage.

[ljharb]

ESM surely needs some sort of import.meta.resolve.

[oleersoy]

They can contain both source location and content by using "sourcesContent". What is missing?

@bmeck Do you have an example of a typescript repository setup that has this enabled so that we can see the process?

I think in general people take the path of least resistance. The simplest path to publish to NPM is to specify a base directory. Everything else is more work. So the short answer the What is missing question is Simplicity.

The longer answer is unit tests, access to the original source the way it was written ... can we add logging statements using the source map?

Is this burden 15 minutes everytime you publish? Or could it be a project bootstrap like so many other workflows use. I would think this is minimal since all the steps described above could be automated.

So we are devs. Once we get used to a certain flow the setup becomes trivial. It could be a project bootstrap, but as you have probably noticed the approach to project layout in the Javascript world is not the most standardized in the world. Everyone has their own favorite approach to doing things. I for example use a simple createdist.js script ... others use gulp ... others ... etc. etc. etc. All of this adds cost because we can almost never count on a standardized structure when adopting a project off of NPM. Anything we can do to simplify this would be of UGE value because 15 minutes for me ... is just 15 minutes, but 15 minutes across all the developers that develop for Node is a lot of minutes ... plus there no shortage of tech to get up to speed on ... the less there is to learn the better.

This seems odd to me, why do you want the github repository for an installed package instead of the interface that the package seeks to ship (via files)?

It's nice to be able to read up on a a project in the github repository or in VSCode and see the same picture. It's even nicer to be able to change the project directly in the node_modules/project directory when attempting to debug an issue. The absolute shortest path to doing this is having a mirror copy of the project the way it exists on Github in the node_modules folder.

How are they doing this? I ask because in general I would not support editing your node_modules as a strong workflow that we should prioritize. In fact, it is something I would personally discourage because it means that things in node_modules are more fragile due to ecosystem usage if we encourage that.

Again people are lazy (Or seek the path of least resistance). If devs can look in the node_modules directory and see the same source they have been looking at on github, then they are more likely to contribute rapid fixes and have better communication with the owner of the repository.

If we make them jump through 3 more hoops to get the setup they need to properly test something, then this is less likely to occur, and the net effect of that on the entire NPM ecosystem is sizable.

[YuriGor]

Hi folks, I am looking for a way to have UMD/CommonJS/ES Modules builds in the same package, and still allow cherry pick of individual methods.

While the first part solved by using browser, main and module properties of the package.json so the user can just import or require the full version of the library using package name, cherry-pick still doesn't work, because it tries to require a file with the name of the method in the root of the project, not in the dist/cjs folder where "main" property aimed.

So as I got it - now there is no way to declare base dir for the different case?

It would be nice to allow an object value to be assigned to browser, main and module properties, where directory and entry point file name will be specified separately.

like this:

{
  "main": {
    "dir":"dist/cjs/",
    "name":"deepdash.js"
  },
  "module":{
    "dir":"dist/esm/",
    "name":"deepdash.js"
  },
  "browser": "dist/umd/deepdash.min.js"
}

this way require('deepdash') will be resolved to dist/cjs/deepdash.js (main.dir+main.name) require('deepdash/filterDeep') will be resolved to dist/cjs/filterDeep.js (main.dir+/filterDeep) and same for import "module"

[iamrenejr]

Jumping on this even though it's a year after the main discussion happened. Have the core maintainers changed their mind about this? I also think there is immense value here and it's worth continuing the discussion.

If a consumer of your package decides they want to import, say, your package.json with require('mypackage/package.json'), wouldn't this additional layer of indirection make things unnecessarily difficult and confusing?

No you would still do that exactly the same way you are doing it now.

That doesn't make any sense. How is require('mypackage/package.json') supposed to access the file in the directory ./node_modules/mypackage/package.json if node is internally using the value of root: './src' in that package.json file to resolve require.resolve('mypackage/mypackage.json') to ${process.cwd()}/node_modules/mypackage/src/package.json (or however that works, you get what I'm saying)? Either it accesses mypackage/src/package.json or mypackage/package.json, and if you're saying it's the latter, that's really inconsistent behavior.

What if require('mypackage/package.json') resolves to ./node_modules/mypackage/package.json in the absence of root: './src', but resolves to ./node_modules/mypackage/src/package.json once root: './src' is added, as you would intuitively expect? Then a special syntax is set to always point to ./node_modules/mypackage/package.json in the presence or absence of root being specified. For example, require('mypackage/$$package.json') will always give you ./node_modules/mypackage/package.json. It doesn't have to be that, it could be something more obscure to prevent colliding with existing files that might already start with $$. Maybe ++package.json, or #!package.json... could be anything.

After all, the addition of a root property (or base, or projectRoot, whatever it might be called) is something all npm package maintainers need to do manually if this feature is added. If there is any code that references the package.json of a dependency, they would also manually update that to specify that they want the project root (ie change it to require $$package.json). This syntax would go for any file or directory they want to access from the root of the project.

And this way, it is opt-in and wouldn't break anything unless package developers wanted to use the new syntax and manually rolled out the changes themselves.

[rbuckton]

The SystemJS loader has a feature similar to this in the form of Import Maps.

TypeScript has a feature like this in the form of Path Mappings, though its purely for design-time module resolution when working with complex build systems and doesn't affect runtime behavior

If package.json supported something like the Path Mappings approach from TS, you could define something like this:

{
  ...
  "paths": {
    "api/*": ["dist/api/*"],
    "*": ["*", "dist/*"]
  }
}

Then require("mypackage/package.json") could still map to ./node_modules/mypackage/package.json, but require("mypackage/foo") could map to ./node_modules/mypackage/dist/foo.js and require("mypackage/api/bar") could map to ./node_modules/mypackage/dist/api/bar.js.

[devsnek]

moving into the future, this seems like a good case for loaders. We try to avoid adding stuff to package.json because it really slows everything down.

[jkrems]

@rbuckton The latest node (12.9) does support something similar although it's still behind the flag --experimental-exports:

{
  "name": "pkg",
  /* [...] */
  "exports": {
    "./foo": "./target.js",
    "./bar/": "./dist/nested/dir/"
  }
}

Now require('pkg/foo') resolves to target.js and specifiers starting with pkg/bar/ will be redirected into dist/nested/dir/*. But this is by design one single mapping that shouldn't require looking at specific files to evaluate. Given that this indirection already adds confusion, we didn't want to have yet-another chain of possible results that require browsing directories to figure out. It also happens to map fairly nicely to import maps which isn't a coincidence.

[thw0rted]

It seems like the exports-map neatly solves this problem. Is there any danger of it going away in the future? As far as I can tell it addresses all the concerns addressed here and in #14970 with no particular downsides.

[jkrems]

The syntax I posted above should be fairly stable. I wouldn’t expect exports to go away completely at this point. What may still change while are details about the array/conditional exports syntax. But path mappings should stay the way they are.

[MylesBorins]

Closing since exports maps landed in 13.x 🎉

[alshdavid]

Is there a way to polyfill this for older node versions? Can you patch the require builtin to include resolution of the exports?

node -r exports-patch/register index.js