search

nix-community / yarn2nix

Generate nix expressions from a yarn.lock file [maintainer=???]
GNU General Public License v3.0
123 stars 61 forks source link

Fails to install transitive Git dependencies #127

Open thomashoneyman opened 4 years ago

thomashoneyman commented 4 years ago

Short summary

If you have a dependency which has a transitive dependency that points to a GitHub URL, then Yarn and NPM will successfully install it, but yarn2nix will fail with an SSL error when trying to fetch that transitive dependency.

For example, the npm-license-crawler package relies a fork of license-checker which points at a GitHub URL. This dependency list in package.json fails:

  "dependencies": {
    "npm-license-crawler": "0.1.9"
  }

However, you can work around this issue by adding the transitive dependency directly into the package.json file. Doing this does cause small changes in the generated lock file, namely that the .git extension is trimmed from the key and resolved url for the license-checker package, which may be all that's needed to work correctly.

Therefore this dependency list works:

  "dependencies": {
    "npm-license-crawler": "0.1.9",
    "license-checker": "git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893"
  }

Full reproduction + steps taken to fix

You can exactly reproduce this error with the below package.json file, default.nix file, and the yarn executable in a shell so you can generate the yarn.lock file by running yarn.

{
  "name": "my-yarn-package",
  "version": "1.0.0",
  "dependencies": {
    "npm-license-crawler": "0.1.9"
  }
}
let
  pkgs = import pinned-nixpkgs { };

  pinned-nixpkgs = builtins.fetchTarball {
    name = "nixos-18.09";
    url = "https://github.com/nixos/nixpkgs-channels/archive/a7e559a5504572008567383c3dc8e142fa7a8633.tar.gz";
    sha256 = "16j95q58kkc69lfgpjkj76gw5sx8rcxwi3civm0mlfaxxyw9gzp6";
  };

  # Nov 22, 2019
  yarn2nix = import (pkgs.fetchFromGitHub {
    owner = "moretea";
    repo = "yarn2nix";
    rev = "9e7279edde2a4e0f5ec04c53f5cd64440a27a1ae";
    sha256 = "0zz2lrwn3y3rb8gzaiwxgz02dvy3s552zc70zvfqc0zh5dhydgn7";
  }) { inherit pkgs; };

in

yarn2nix.mkYarnPackage {
  name = "my-yarn-package";
  src = ./.;
}

Attempting to build this tiny project will fail with an SSL certificate problem:

λ nix-build --keep-failed --show-trace
these derivations will be built:
  /nix/store/r3qg57qxjpqm607xs83cx4mvd78w1bpv-my-yarn-package-modules-1.0.0.drv
  /nix/store/ybn6lc6l2l75r899slq8p10b06q6dvvm-my-yarn-package.drv

...

[1/4] Resolving packages...
error Command failed.
Exit code: 128
Command: git
Arguments: ls-remote --tags --heads https://github.com/mwittig/license-checker
Directory: /private/var/folders/bs/y8pdflhj6t9c3963clpgfz_m0000gn/T/nix-build-my-yarn-package-modules-1.0.0.drv-4
Output:
fatal: unable to access 'https://github.com/mwittig/license-checker/': SSL certificate problem: unable to get local issuer certificate

So the git command fails with an SSL certificate problem when attempting to access a remote URL, namely the mwittig/license-checker repository. This dependency is not listed in the package.json, but it’s a transitive dependency of npm-license-crawler, which we can see in the yarn.lock file:

npm-license-crawler@0.1.9:
  version "0.1.9"
  resolved "https://registry.yarnpkg.com/npm-license-crawler/-/npm-license-crawler-0.1.9.tgz#2e92f554ef38a20d377a2fbdae3bc7bd9440e8d5"
  integrity sha1-LpL1VO84og03ei+9rjvHvZRA6NU=
  dependencies:
    async "^2.6.1"
    jquery-extend "~2.0.3"
    license-checker "git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893"
    mkdirp "^0.5.1"
    nopt "^3.0.6"
    nopt-defaults "^0.0.1"
    nopt-usage "^0.1.0"
    treeify "^1.1.0"

"license-checker@git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893":
  version "1.0.0"
  resolved "git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893"
  dependencies:
    chalk "~0.5.1"
    mkdirp "^0.3.5"
    nopt "^2.2.0"
    read-installed "~3.1.3"
    treeify "^1.0.1"

Fix 1: Add the transitive dependency to package.json

The first thing I noticed is that it’s a little strange npm-license-crawler sees the license-checker dependency at this URL, which does not include the .git extension:

git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893

But that license-checker lists itself at this url, which does include the .git extension:

git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893

I tried putting these URLs directly into the package.json file to see which one causes the problem, and found to my surprise that putting either URL into the package.json file will change the resulting yarn.lock file and will cause the Nix build to succeed.

First, the diff of putting the URL that npm-license-crawler wants, which has no extension:

diff --git a/src/package.json b/src/package.json
index b48b4e1..c465d79 100644
--- a/src/package.json
+++ b/src/package.json
@@ -2,6 +2,7 @@
   "name": "my-yarn-package",
   "version": "1.0.0",
   "dependencies": {
-    "npm-license-crawler": "0.1.9"
+    "npm-license-crawler": "0.1.9",
+    "license-checker": "git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893"
   }
 }
diff --git a/src/yarn.lock b/src/yarn.lock
index f942ffe..11758dc 100644
--- a/src/yarn.lock
+++ b/src/yarn.lock
@@ -146,9 +146,9 @@ json-parse-helpfulerror@^1.0.2:
   dependencies:
     jju "^1.1.0"

-"license-checker@git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893":
+"license-checker@git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893":
   version "1.0.0"
-  resolved "git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893"
+  resolved "git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893"
   dependencies:
     chalk "~0.5.1"
     mkdirp "^0.3.5"

This removes the .git extension on the resolved URL and the fetch now succeeds; that makes sense, because the URL I manually provided in the package.json didn’t include it either, and because npm-license-crawler listed this key as the resolver for the license-checker dependency. Maybe now a lookup is succeeding.

What about including the extension in the package.json file, which is how license-checker resolves in the broken version?

diff --git a/src/package.json b/src/package.json
index b48b4e1..7763c23 100644
--- a/src/package.json
+++ b/src/package.json
@@ -2,6 +2,7 @@
   "name": "my-yarn-package",
   "version": "1.0.0",
   "dependencies": {
-    "npm-license-crawler": "0.1.9"
+    "npm-license-crawler": "0.1.9",
+    "license-checker": "git+https://github.com/mwittig/license-checker.git#d546e3f738e14c62e732346fa355162d46700893"
   }
 }
diff --git a/src/yarn.lock b/src/yarn.lock
index f942ffe..b22fc51 100644
--- a/src/yarn.lock
+++ b/src/yarn.lock
@@ -146,6 +146,16 @@ json-parse-helpfulerror@^1.0.2:
   dependencies:
     jju "^1.1.0"

+"license-checker@git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893":
+  version "1.0.0"
+  resolved "git+https://github.com/mwittig/license-checker#d546e3f738e14c62e732346fa355162d46700893"
+  dependencies:
+    chalk "~0.5.1"
+    mkdirp "^0.3.5"
+    nopt "^2.2.0"
+    read-installed "~3.1.3"
+    treeify "^1.0.1"

This leaves the old entry for license-checker in place, with the .git extension, and it also adds a new entry for license-checker without the .git extension. This nix-build also succeeds!

Moral of the story so far: so long as you put the GitHub dependency directly in the package.json file, with or without a .git extension, the lock file will generate a key and resolved url which does not include the .git extension and which succeeds with yarn2nix. So one (possible) solution to this is to take any failing GitHub transitive dependency and add it directly to the package.json file.

Failed Fix 2: Provide certificates to Yarn when fetching

Ideally this would just work without adding the transitive dependency to the package.json file, so I wanted to check on this SSL error. If I run nix-build --keep-failed then I can see the build at the moment it failed.

One thing stood out to me: the build is explicitly unsetting the SSL_CERT_FILE environment variable,.

$ cat .env-vars | grep "SSL" 
declare -x SSL_CERT_FILE="/no-cert-file.crt"

I chased this for a bit, providing pkgs.cacert to Yarn as a build init in an overlay, and that got me farther, but the process just hangs at “fetching packages…”. I didn’t dig any further as I suspect this issue is falling out of the prior information I’ve demonstrated.