Changes to avoid rebuilds of `pkgConfig.postInstall` code on each `nix-build`

[Ma27]

Note: @lheckemann and I developed this for a customer project, so please keep in mind that this is currently far from ready and has some downsides/regressions I'll mention below.

However, this approach works surprisingly well, so we decided to file a (draft)-PR against this repository to discuss if this is integratable into the upstream yarn2nix. We are well-aware though that getting this mergable is not an easy task, so we are already "prepared" to maintain those changes in transloadit/yarn2nix if needed.

Also, when reviewing, please look at the changes except for https://github.com/nix-community/yarn2nix/commit/9694f004660be952511d2206af32dce7a1e96d86 (which basically re-adds the Nix code of yarn2nix to the repository). This is needed since the yarn2nix subtree in nixpkgs actually contains all yarn2nix sources, but has a dev script to sync that code with nix-community/yarn2nix, so we decided to develop in a fork of this repository.

---

tl;dr

Usually an expression like this is used when building a yarn project with dependencies that have e.g. C bindings:

with import <nixpkgs> {};
let
  nodeHeaders = fetchzip {
    name = "node-v${nodejs.version}-headers";
    url = "https://nodejs.org/download/release/v${nodejs.version}/node-v${nodejs.version}-headers.tar.gz";
    sha256 = "sha256-sASVCHD8e9P1VPOVLBHpY5wuyhBJvO0qe9bUDj8k3UU=";
  };
in mkYarnModules {
  name = "foo";
  pname = "foo";
  version = "0.0.0";
  packageJSON = ./package.json;
  yarnLock = ./yarn.lock;
  yarnNix = ./yarn.nix;
  pkgConfig.sharp = {
    buildInputs = [ nodePackages.node-gyp python3 pkgconfig vips ];
    postInstall = ''
      node-gyp --nodedir=${nodeHeaders} rebuild
    '';
  };
}

This kind of expression has the downside however that you always need to rebuild the C bindings which slows down CI and dev cycles (if yarn2nix is used for development).

These changes basically move the build scripts defined in pkgConfig into their own derivations and ensure that all required npm dependencies are available as well. Due to that, the time for a recompile is saved.

Downsides

In the current state of the branch, the following features are not available:

• Right now it's always necessary to generate the yarn.nix before running nix-build. This is needed since we have to fetch metadata about transitive dependencies from either the registry itself or the package.json of a git repository. That kind of information is needed to make sure that only needed dependencies are available in a derivation to build a NPM package with an extra build script in pkgConfig.*.

• I had to change the format of yarn.nix a bit, so existing files would break.

[lheckemann]

We should probably fix the tests ^^

[Ma27]

We should probably fix the tests ^^

I'd like to get some feedback first: also, I removed some features here, so this needs some more discussion before getting the testsuite running again.

[Ma27]

cc @lassulus should I base this off nixpkgs then (re https://github.com/NixOS/nixpkgs/pull/108138)?