node-packages' generate.sh makes many duplicate requests

chreekat commented 5 years ago

Issue description

I ran generate.sh as per Nixpkgs manual §9.11, and before it had even finished I noticed a great number of duplicate requests to npmjs.org.

Since I'm not certain about npm's inner workings, I used tcpdump to confirm that duplicate HTTP requests are being sent.

Some stats

74% of all requests are duplicates.

Note: all the below is only v10 packages, and I canceled the script before it even ran to completion. (I'm running it again just to confirm.)

Edit: generate.sh fails on master@f0ec7c6f072e0c734d95d3a62344c949faa3443a right now anyway, but I still get the following stats:

$ grep '^http 200' reqs | wc
   5881   17643  274467
$ grep '^http 200' reqs | sort -u |wc
   1541    4623   72882

(5881.0 - 1541)/5881 ~ .74 : 74% of requests are duplicates.

An example

$ grep '^http 200.*ansi-regex' reqs
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex
http 200 http://registry.npmjs.org/ansi-regex

Steps to reproduce

nixpkgs $ cd pkgs/development/node-packages
node-packages $ nix-shell -p nodePackages.node2nix
nix-shell $ node2nix --nodejs-10 -i node-packages-v10.json \
    -o node-packages-v10.nix  -c composition-v10.nix |& tee reqs

Technical details

system: "x86_64-linux"
host os: Linux 4.14.114, NixOS, 18.09.2543.41245517a98 (Jellyfish)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.1.3
channels(root): "nixos-18.09.2543.41245517a98"
channels(b): "unstable-19.09pre173349.07b42ccf2de"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

dtzWill commented 5 years ago

I've noticed this too, it has been this way for some time :/. I've thought about using a caching proxy or something as a kludge-fix, but really it'd be best to avoid all the duplicate work altogether (and I'm unsure how much a role the request plays vs other work).

dtzWill commented 5 years ago

Last time I looked at this, I got as far as "it's uses javascript" (not my strongest :3), "it caches lookups based on nix AST of the expression evaluated", dunno if I found if AST's were different (perhaps in unimportant ways) or if there was a bug in the caching. Oh, and because javascript is confusing :P it might be the case that the cache works just fine but is limited to each invocation, and we're invoking it for each package or something?

Haha, hope this helps! Fixing this would help a lot, and facilitate more regular updating.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

NixOS / nixpkgs