Caching of packages built by node-gyp?

[zarybnicky]

I tried out this experiment in Node+Nix packaging and I have a few things to note while the process is fresh in my mind:

buildNodeModules.hooks.linkNodeModulesHook is a setup hook, which is good for an already set-up project. It is less good for 1) setting a project up, where a linkNodeModules script (or even linkNodeModules --force-overwrite) would be useful 2) tools like devenv, which for some reason don't like setup hooks (their custom mkNakedShell), but only allow a single enterShell script, where you could add linkNodeModules

The entire node_modules is built as a single derivation (unless I'm missing something?). That works quite well for JS-only projects, but as soon as a native dependency comes into play, it is rebuilt every single time, due to the npmConfigHook. Coming from yarnpnp2nix where each dependency has its own derivation, this was quite surprising.

npm i --package-lock-only is a must, which I only discovered from the npmlock2nix issue linked from readme - npm i --package-lock-only <package> && direnv reload && <wait while node-gyp rebuilds everything> is an approximation of the loop that I had.

To be specific, I saw the release announcement just in time when I was starting a new project that needed nodegit which requires libgit2 bindings - for which I would need a FHS, npmlock2nix, or Docker. I was able to get a working development setup with the following Flake (omitting devenv/overmind and other irrelevant parts). I'll probably stick with it for now and see how it works out as the project grows.

{
  inputs.nixpkgs.url = github:NixOS/nixpkgs/release-23.05;
  inputs.buildNodeModules-flake.url = github:adisbladis/buildNodeModules;
  inputs.buildNodeModules-flake.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, devenv, buildNodeModules-flake, ... } @ inputs: let
    pkgs = import nixpkgs {
      system = "x86_64-linux";
    };

    buildNodeModules = buildNodeModules-flake.lib.x86_64-linux;

    modules = buildNodeModules.buildNodeModules {
      packageRoot = ./.;
      nodejs = pkgs.nodejs;
    };

    nodePackage = pkgs.stdenv.mkDerivation {
      pname = "codestats";
      version = "1.0.0";
      src = ./.;

      nativeBuildInputs = with pkgs; [
        buildNodeModules.hooks.npmConfigHook
        libkrb5
        inetutils
        file
        gccStdenv.cc
        curl
        openssl
        nodejs
        nodejs.passthru.python # for node-gyp
        # npmHooks.npmBuildHook
        npmHooks.npmInstallHook
      ];

      nodeModules = buildNodeModules.fetchNodeModules {
        packageRoot = ./.;
      };
    };
  in {
    devShells.x86_64-linux.nodeModules = pkgs.mkShell {
      buildInputs = [
        buildNodeModules.hooks.linkNodeModulesHook
      ];

      nodeModules = buildNodeModules.buildNodeModules {
        packageRoot = ./.;
        inherit (pkgs) nodejs;

        buildInputs = with pkgs; [
          libkrb5
          inetutils
          file
          gccStdenv.cc
          curl
          openssl
          nodejs
          nodejs.passthru.python # for node-gyp
        ];
      };
    };
  };
}

[adisbladis]

The function buildNodeModules is already designed in such a way to reduce rebuilding of node_modules: Sources are not passed to the build, only package.json & package-lock.json. So the interactive development use case should be cached most of the time.

The entire node_modules is built as a single derivation (unless I'm missing something?). That works quite well for JS-only projects, but as soon as a native dependency comes into play, it is rebuilt every single time, due to the npmConfigHook. Coming from yarnpnp2nix where each dependency has its own derivation, this was quite surprising.

Yes, that's an unfortunate side effect of how prevalent circular dependencies are in nodejs. Most node2nix solutions ends up with a single derivation for node_modules to just not have to deal with dependency cycles.

buildNodeModules.hooks.linkNodeModulesHook is a setup hook, which is good for an already set-up project. It is less good for 1) setting a project up, where a linkNodeModules script (or even linkNodeModules --force-overwrite) would be useful 2) tools like devenv, which for some reason don't like setup hooks (their custom mkNakedShell), but only allow a single enterShell script, where you could add linkNodeModules

Apparently https://github.com/numtide/devshell tries to minimize environment, and one of the methods to achieve that is to not run setup hooks. That's fine if all you care about is a few tools in $PATH, but for anything more advanced that model is going to break. I'd consider this an issue with devShell.

[zarybnicky]

Thanks for responding. Since this was mostly meant as an experience report, close this issue as is: I'll expose linkNodeModules as a script myself, try to live with the node-gyp rebuilds, and perhaps revisit sometimes in the future.