A more stable method of retrieving package information

[bobvanderlinden]

In #21 there was a discussion about an alternative method of retrieving package information.

At the moment npm and node change quite often, as do their file structures. Relying on the structure of package.json is therefore often errorprone.

If we could reuse npm to do as much as possible, we could circumvent this problem. It might be somewhat less efficient compared to the current implementation, but since npm2nix is usually not run daily, this does not matter much.

I created this issue to focus on reusing npm and gather all the necessairy bits needed to get to an implementation of npm2nix.

• [x] npm install creates node_modules with all dependencies of package.json installed.
• [x] npm ls --json shows a recursive list in JSON of all packages and their dependencies.
• [x] Tarballs from npmjs: The cache that is normally in ~/.npm/{packageName}/{packageVersion}/package.tgz can be used to generate hashes for the .tgz files. Alternatively, we can use the shasum from npmjs registry.
• [x] We can make npm use a different cache directory so that it doesn't download files into $HOME.
• [ ] Tarball URLs: I'm not sure these are placed in ~/.npm/
• [ ] Git URLs: We need a way to hash these. The current method in npm2nix could be used, but it is desirable to let npm retrieve the git repo.
• [ ] Custom URLs: github: and probably others will be used as dependencies in the near future. We need to get npm to download these as git repos, so that we can hash them.
• [ ] Handle directory structure for github: URLs.

[offlinehacker]

There's one other issue, the format of dependencies github:owner/name produces different directory layout in node_packages(owner/name instead of only name), i also have to handle this in nix.

[bobvanderlinden]

Ouch, I wonder how this works for npm ls.

[bobvanderlinden]

So I dug a little again, but isn't npm shrinkwrap what we want?

For instance:

cd popcorntime
npm install
npm shrinkwrap
cp npm-shrinkwrap.json nixpkgs/pkgs/applications/video/popcorntime/

Inside fetchnpm.nix:

stdenv.mkDerivation {
  builder = ''
    cp ${./npm-shrinkwrap.json} ./
    npm install
    cp -r node_modules $out
  '';
  outputHash = "...";
}

In default.nix link node_modules to the result of fetchnpm. EDIT: I'm also wondering whether the result of installing shrinkwrapped dependencies will always result in the same node_modules (same hash). Do you know, or is this something to just try? It's just an example, but is this a viable direction?

[bobvanderlinden]

I did an experiment with npm-shrinkwrap. I ran npm shrinkwrap in popcorntime's repository and used the resulting file in a nix-expression where it runs npm install. There was some hackery, but it seems to be creating the same node_modules directory each time it is called here.

See here: https://gist.github.com/bobvanderlinden/05cb8e4b4400618ee487

[wmertens]

The problem with shrinkwrap is that it uses the currently installed modules. If you run it against a fresh source without node_modules, it is empty.

Basically we need something that gets a package.json and dumps out the resolution. This resolution is the description nix needs to build the packages.

Then each of the packages in the resolution can be built, using the resolved dependencies. To get around the symlinked-dependencies-use-real-location-for-dependencies problem simply symlink one level down.

[bobvanderlinden]

It might be a side-effect, but isn't the result of npm install && npm shrinkwrap the resolution you're looking for? It outputs a shrinkwrap file that contains the package names and versions of that time. Npm knows where to get those exact files. If that ran when you're packaging an application and include the shrinkwrap file in the nix package, it'll result in the exact same node_modules contents. A hash for the resulting node_modules directory will assert it is indeed always the same (it should be because that is what shrinkwrap is meant for).

Additionally, in an ideal situation the devs of an application itself include their version of the shrinkwrap file upon releasing stable versions. This can be reused, so that the package will include the exact packages the developers intended the application to use.

It's probably less space efficient and cost more time building than using npm2nix, but it might be more practical for maintaining some huge and complex packages.

[dzmitry-lahoda]

may be it can be closed?

[bobvanderlinden]

I haven't used npm2nix for quite a while, I'll close it. 👍