why doesn't the npm registry leverage the npm client?

[jhoguet]

I have been experimenting with jspm for several months now with the hopes of evangelizing it to a wider audience at my company, but there are a few concerns that are holding me back, this is one of them

why doesn't the npm registry leverage the npm client?

part of the appeal of using npm registry for my client side packages, is that I understand and use the tooling (npm) that supports it. With the npm registry I am surprised to see that it is actually making http requests and not leveraging the cli at all.

What were the motivations that led to this decision? If this is feasible, are you open to a pull request? Any pointers?

thanks so much!

[lookfirst]

Ha! I've been saying that for ages now... even started work on a proof of concept... https://github.com/lookfirst/honk

[jhoguet]

@lookfirst any ideas on what requirements led to the npm client not being feasible?

[guybedford]

@jhoguet this exact discussion is somewhere in the issues, but I can't find it right now. The npm client is a lot of code to have a dependency. jspm has its own style of version resolution, which could wrap the npm client just to handle downloads of code, but that would be loading a lot of unnecessary overhead as the download and lookup to the registry itself is just a few lines of code.

Also I don't even know that the npm client provides an API to do such a single-file download. When there is a version conflict, npm will install the extra version in a nested node_modules folder, while jspm retains versions as flat. So we would need such a single-file download encapsulation offered by npm to take advantage of that.

In addition from a security perspective, running npm install will give full code execution permissions during install of whatever you install on your machine. jspm has the nice property that you can install untrusted code on your host machine, and run it in a sandboxed browser, without security fears.

[lookfirst]

My point is that the download and version resolution is a bit more than that with npm. The code isn't simple, otherwise people wouldn't be complaining and npm itself would just be a few lines of code.

You don't need to download the files from npm, you just tell npm which version you want and then let it do all the background magic to put the dependency into the node_modules folder and then pluck the files from there yourself. All the retry logic, caching and auth is handled for you.

The security thing is a red herring bike shed issue. What if every single project that I download off npm is my own repo? Heh.

[jhoguet]

@guybedford I see how npm (the client) brings with it, its own overhead, but it is a system I think jspm users already understand and have accepted. By embracing that, maybe we can end up with less overall complexity. This seems to be the most compelling argument for similar technologies like webpack.

For instance:

1. I already have npm configured for my npm server, so JSPM would not need to deal with the .npmrc logic it currently does.
2. I can already retrieve git packages through npm (git+https/ssh), so the github registry would not be necessary.

There would certainly be complexity to deal with to get these packages ready for web, but that is exactly what I would expect JSPM to do. That complexity is easier to swallow when it is sitting on top of complexity I already live with (the npm client).

Please help me understand the nuances of JSPM that made this not practical...

jspm has its own style of version resolution

What about JSPM's style make it not practical to use the node_modules folder (eg symlinks to them from jspm_packages)? This is likely a lack of JSPM knowledge on my part, so please point me to the appropriate features/ functionality / design.

Also I don't even know that the npm client provides an API to do such a single-file download. When there is a version conflict, npm will install the extra version in a nested node_modules folder, while jspm retains versions as flat. So we would need such a single-file download encapsulation offered by npm to take advantage of that.

Couldn't we just symlink the deep folder that npm created in node_modules to a flat folder structure in jspm_packages? something like public/jspm_packages/npm/something@0.1.0/ -> node_modules/something

running npm install will give full code execution permissions

I agree with @lookfirst I am not following the security argument. My assumption is that JSPM users are using NPM and therefore any security concerns they have with NPM have already been accepted / mitigated.

I appreciate your time to help me understand this, and hope that the end result is a way for me to contribute back.

Aside: I realize the github registry is far more advanced than what can be accomplished via npm git+ssh/https (eg. version resolution), and that there are a lot of packages using the github registry (https://github.com/jspm/registry/blob/master/registry.json). But I think that we could still support all of those packages using npm (client) via git+ssh/https until they publish to npm. If I am right, then we could reduce the complexity of JSPM, which should lead to a better chance of success. It seems that npm is enough for other technologies like webpack.

[lookfirst]

The symlink won't work as there is source file munging that goes on. That said, just coping the files from node_modules would work perfectly.

[jhoguet]

re: installing a single file (package).

I was curious what where does in npm install (api)

so I tried the following

var npm = require('npm');
npm.load(function(){
    npm.commands.install('my_folder', ['webdriverio']);    
})

and it created a node_modules folder in my_folder and put webdriverio in there.

my_folder/
└── node_modules
    └── webdriverio

This still creates the deep hierarchy @guybedford mentioned above, but it shows that if we wanted to, we could install the node modules in our own folder, separate from the one the npm cli typically manages. Not sure this would even be worth it, but good to know.

[lookfirst]

@jhoguet https://github.com/lookfirst/honk/blob/master/src/NpmRepository.js

[jhoguet]

@lookfirst thanks for sharing, did you find anything you couldn't do that jspm is requiring? It looks like you have proven this is possible, what is left to turn this into a jspm registry?

[guybedford]

@jhoguet the very fact that jspm downloads into flat versioned folders, while npm downloads into hierarchical node_modules folders is why we can't share the same download implementation. In addition, npm will attempt to download and cache all dependencies, while jspm will need its own version resolution hooks. There is no way to hook out to npm, without npm downloading incorrect version resolutions than the ones jspm would use, and downloading them to the wrong locations (nested node_modules). It would add processing and network overhead making jspm slow to build on top of this.