Reduce some node packages technical debt

[svanderburg]

Today, I have regenerated all nodePackages to use node2nix 1.8.0 that I have released today. The process was mostly straight forward, but I have noticed that some things are slowly growing out of hand. To reduce this technical debt, I could use some help.

What I'd like to address is the following:

• In addition to the packages in pkgs/development/node-modules/ folder, I also know that we have NPM package sets outside this directory (e.g. base16-builder, airfield etc.). I have updated these as well. However, I also believe that not all of these projects should be isolated -- some of them are isolated for a good reason, e.g. they need specialized settings for deployment such as a supplement JSON file, but some can also easily exist in the shared NPM package set. It would be nice to investigate if there are packages that don't use any specialized settings that we could integrate.
• The size of the shared generated package set keeps growing. As a policy (that you can find in the Nixpkgs manual), I basically stated that we should only add end-user packages or library packages that are needed by non-NPM projects. For some packages that have been added, e.g. textlint-rule* plugins, I think they should not belong there -- they should be deployed as project dependencies, not package dependencies. It would be nice to go over the list and identify these non end-user software packages and try to get rid of them. Perhaps, a friendly way to get rid of them is to ping the committer and ask him/her for the rationale for adding it. Maybe we can suggest an alternative approach to get the underlying requirement covered.
• The generate.sh script no longer works for me. It generalizes a few things, which is good, but it also tries to do the generation in a temp directory (including making a copy of Nixpkgs) and then copying the corresponding expressions back. In my case, it always fails. IMO I don't think we need such a strategy (since you can always check the git diff that you intend to commit), but I'm not against keeping it if it is generally considered useful. Still, I can't use this script anymore myself, so probably somebody needs to take a look at it and fix it.

[teto]

generate.sh would consistenly fail on unreliable network and I getting the original from git is more involving (as in you need to understand more) than just rerunning the script The error comes from a review-induced change but the fix consists in changing cd "$NODE_NIXPKGS_PATH/pkgs/development/node-packages" to cd "$NODE_NIXPKGS_PATH"

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[chasecaleb]

On this topic, it would be nice if there was a way to run generate.sh for a single package. Right now I'm trying to add a package for the first time, and so far I've been waiting about an hour despite being on a fast fiber connection... it's rather offputting.

Also it looks like the script updates every single Node package, which doesn't seem great when I want to make a PR to add one new package, not update the world.

[teto]

I have the same issue, it took hours to complete the update, all that to realize I used the wrong package name and rerun everything. I think this should be part of a goal to unify the different ecosystem updaters. the vim plugin updater can cache the results so subsequent runs dont need to do as much.

[teto]

@svanderburg I am ready to help update the script but I need help to devise a strategy to speed up things. Lots of time seems to be about crawling the npmjs registry ? can't we cache that and give some data straight to node2nix ? seems like the final generation has to be done in one pass by node2nix since it will compute the deps and all so not sure what else I could do (but caching the hundreds of registry pings should already be beneficial). I suppose there is no way to split the package list in batches of 100 and merge them afterwards ?