feat: make overlays from JSON definition in `package.json` file

olebedev commented 1 year ago

Background

Writing Nix expressions for non-Nix folks can be overwhelming. For example, developers from product teams can bump and update Node.js packages quite often, and sometimes they have to deal with Nix expressions like this:

self: super: {
  "@jest/globals@27.0.3" = super."@jest/globals@27.0.3".override {
    doChecks = false; };
  "babel-jest@27.0.2" = super."babel-jest@27.0.2".override
    # Add peer dependency
    (x: { modules = x.modules ++ [ (self."@babel/core@7.14.3") ]; });
  "yargs@16.2.0" = super."yargs@16.2.0".override {
    patches = [
      ./patches/0001_yargs_cve.patch
    ]; };
  "left-pad@1.3.0" = super."left-pad@1.3.0".override {
    src = ./vendor/left-pad; };
}

which is pretty natural for folks who are aware of how to write Nix but for those who are not familiar it can be challenging. Also, I found that the complexity of Nix and its ecosystem is the main barrier when people question if we should adopt Nix more, which I believe shouldn't even be asked.

That is, the main point of this PR is to reduce this burden and let non-Nix folks apply overlays defined in package.json files instead of writing Nix expressions in the Nix language.

The solution

The introduced mechanism is very similar to additional sections of package.json file that different package managers from Node.js ecosystem support. For example, Yarn supports the resolutions section, PNPM supports pnpm.overrides (and more), and so on.

For example, instead of creating an overlay function as a Nix expression as above, the same can be done in package.json file now:

{
  "js2nix": {
    "overlay": {
      "@jest/globals": {
        "doCheck": false
      },
      "babel-jest": {
        "addDependencies": [
          "@babel/core"
        ]
      },
      "yargs": {
        "patches": [
          "./patches/0001_yargs_cve.patch"
        ]
      },
      "left-pad": {
        "src": "./vendor/left-pad"
      }
    }
  }
}

In this PR

This PR required many things to be moved around to make it work and a couple of those are naturally aligned with the point of this PR, here are the technical pieces of the changeset:

The js2nix section inf package.json file, and an overlay function generation, see ./overlay.nix file.
The __meta__ section in the resulting generated Nix expression, to make the overlay generation logic work. Otherwise it won't be possible because it's not possible to inspect the whole resulting attrset from within an overlay, it will reach infinite recursion.
Short package aliases. It's possible to get packages from the closure by their name without version or version range, as it was before. That is before it was only possible to get a package from closure by:
```
tree."yargs@16.2.0"
```
It can be gotten by:
```
tree."yargs"
```
(js2nix will abort the evaluation phase if there is a name clash)
Re-formatting of the lib.nix file, which is the cause of a huge diff
The check phase implementation for every buildable package (i.e. if it's not hosted). Now every package that exports either index.js file in the package root folder or the main field in package.json file will be checked by default if it can be required in Node.js context, like require('yargs'). Can be disabled by overriding.
The buildEnv function as the main API entry point to reduce cognitive overloading and make use of js2nix simpler. For example, it's only single function invocation now
```
let
env = js2nix.buildEnv {
  package-json = ./pacakge.json;
  yarn-lock = ./yarn.lock;
};
in env.pkgs.jest # or env.nodeModules
```
See the documentation's addition, for more details.

olebedev commented 1 year ago

is this a breaking change btw?

@pathway27, no, it's not a breaking change, everything will continue working as it was before.

olebedev commented 1 year ago

@pathway27, I will rebase on https://github.com/canva-public/js2nix/pull/7 after it's merged so we can see a regression if any.

canva-public / js2nix