services.node-red.withNpmAndGcc does not work as expected, project support

[serpent213]

Describe the bug

Even with withNpmAndGcc I was not able to install node-red-contrib-modbus via the palette manager. Result was:

13 Jan 04:53:24 - [info] Modul wird installiert: node-red-contrib-modbus, Version: 5.27.2
13 Jan 04:53:35 - [warn] Installation des Moduls node-red-contrib-modbus ist fehlgeschlagen:
13 Jan 04:53:35 - [warn] ------------------------------------------
13 Jan 04:53:35 - [warn] npm WARN config production Use `--omit=dev` instead.
npm WARN deprecated vm2@3.9.19: The library contains critical security issues and should not be used for production! The maintenance of the project has been discontinued. Consider migrating your code to isolated-vm.
npm ERR! code ENOENT
npm ERR! syscall spawn sh
npm ERR! path /persist/node-red/node_modules/serialport/node_modules/@serialport/bindings-cpp
npm ERR! errno -2
npm ERR! enoent spawn sh ENOENT
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! A complete log of this run can be found in: /persist/node-red/.npm/_logs/2024-01-13T03_53_25_156Z-debug-0.log
13 Jan 04:53:35 - [warn] ------------------------------------------
Error: Installation fehlgeschlagen
    at /nix/store/3bn3fgd1yqwf4rnpff6f0b954vlfxrra-node-red-3.1.0/lib/node_modules/node-red/node_modules/@node-red/registry/lib/installer.js:285:25
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
13 Jan 04:53:35 - [error] Error: Installation fehlgeschlagen

Steps To Reproduce

Steps to reproduce the behavior:

1. Setup node-red service with withNpmAndGcc enabled
2. Open web interface, palette manager, install node-red-contrib-modbus

Running nixpkgs-23.11, tried with nodejs_18 and nodejs_20.

Expected behavior

Clean install of the module. The palette manager works fine as is for node-red-dashboard, for example.

Additional context

I was able to solve it with an override:

    (final: prev:
    {
      node-red = prev.nodePackages.node-red.overrideAttrs (oldAttrs: {
        nativeBuildInputs = oldAttrs.nativeBuildInputs or [] ++ [ prev.makeWrapper ];
        postInstall = oldAttrs.postInstall or "" + ''
          mkdir -p $out/bin
          wrapProgram $out/bin/node-red \
            --prefix PATH : "${prev.lib.makeBinPath [
              # module installation
              final.nodejs
              final.nodePackages.npm
              final.bash
              final.gcc
              # projects
              final.git
              final.openssl
              final.openssh ]}"
        '';
      });
    })

I would rename withNpmAndGcc to withBuildEnv (and have it include nodejs and bash) and add an additional option withProjects to provide deps for the new NodeRED project manager – but my Nix fu is reaching its limits here, to implement this cleanly.

Notify maintainers

@pbsds @emilylange @chewblacka @h7x4

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.71, NixOS, 23.11 (Tapir), 23.11.20240108.6723fa4`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

---

Add a :+1: reaction to issues you find important.

[pbsds]

I think none of the mentioned contributors are familiar with the package. Via #116578: @MatthewCroughan @nixinator

[MatthewCroughan]

The correct approach is packaging the nodes purely with dream2nix or node2nix, then symlinking that pure, derivation based node_modules folder into the path for node-red. Anything else is just bound to run into dependency issues. You can't just give NPM everything it could possibly want or need at runtime.

I'm doing something like that here https://github.com/nix-how/ming

Firstly, I'm reproducing a node using dream2nix:

https://github.com/nix-how/ming/blob/0a91236e7c90dfdd8b306f41b9db89d01fb343c0/flake.nix#L34-L40

Then I symlink the resulting node_modules folder into the node-red path

https://github.com/nix-how/ming/blob/0a91236e7c90dfdd8b306f41b9db89d01fb343c0/vm.nix#L90-L92

[MatthewCroughan]

I should also note that nodes are just NPM packages that can be packaged as part of nodePackages in Nixpkgs, and if they were this would mean you could easily include them in a list.

So if you can put https://flows.nodered.org/node/node-red-contrib-modbus into nodePackages I can make a module option for node-red that looks like

{
  services.node-red = {
    enable = true;
    nodes = [  nodePackages.node-red-contrib-modbus ];
  };
}

I prefer the dream2nix approach described above though, so I don't think I'll spend any time on this since what I've described above is easy to implement in your own system. Maintaining the node-red nodes as part of nodePackages in Nixpkgs will likely be a maintenance burden compared to calling a single dream2nix function in your own system flake.

[serpent213]

Surely it's preferable to have the modules available as proper Nix packages. OTOH, I see bash and node as pretty reasonable components of a node build environment – and they are installed anyway, so it's just a few more links.

What do you think about the project support? Those deps stem from NodeRED itself.

[MatthewCroughan]

It's more complicated to let NPM manage the packages than to just use Nix. Just make a derivation for the package and then pass it in.

[serpent213]

Ok, I think the nodes option would be a good idea. Still, how would you deal with the project feature and its deps?

[MatthewCroughan]

Still, how would you deal with the project feature and its deps?

Oh yeah, I played with that a bit hoping it would be good, but I found that it was poorly implemented and you're better off making a systemd service that does a git commit and push on the service outside of node-red. I find that node-red is really good on its own, with the core nodes, and a few manually packaged nodes.

Giving node-red access to NPM means you can try to install any unreproducible node, and potentially break your installation. Giving node-red access to git and ssh-keygen results in key management issues that are complex to solve with nix/jq/systemd since it doesn't give you methods of providing the keys and tries to do everything itself. The git interface is not complete or worth using IMO.

If you want to do these things, all you have to do is add do the path like you've already suggested in the original post, and this overriding/overlay ability via the Nix expression language is what Nix is good at.

An easier way than patching the PATH of the package, for your own configuration is like this:

{ pkgs, ... }:
{
  systemd.services.node-red.path = with pkgs; [ git ssh-keygen ];
}

Adding these deps to the path in Nixpkgs just means we have more untested/untestable features that people might make issues about. And I know from experience with node-red that these features aren't that great.

[MatthewCroughan]

You could also make it an argument on the node-red package, that would make the most sense. Then you could have:

{
  services.node-red.package = (node-red.override { withProjectsFeature = true; });
}

[serpent213]

Oh yeah, I played with that a bit hoping it would be good, but I found that it was poorly implemented and you're better off making a systemd service that does a git commit and push on the service outside of node-red. I find that node-red is really good on its own, with the core nodes, and a few manually packaged nodes.

Giving node-red access to NPM means you can try to install any unreproducible node, and potentially break your installation. Giving node-red access to git and ssh-keygen results in key management issues that are complex to solve with nix/jq/systemd since it doesn't give you methods of providing the keys and tries to do everything itself. The git interface is not complete or worth using IMO.

I agree, “projects” is basically still beta. I'm pretty new to Nix (and NodeRED) and I like to keep my little unreproducible, mutable corners – especially for exploring a technology and various modules in this case. It's just not much fun to poke around declarations for half an hour for a quick test. On a production system it's different, of course, but I feel it's nice to keep both ways open (for Nix in general).

Closing this issue now, thank you for your helpful input!

P.S.: Still I think

I would rename withNpmAndGcc to withBuildEnv (and have it include nodejs and bash)

would be a good idea. 😌

[serpent213]

@MatthewCroughan

dream2nix does not seem to work for me, there's basically no documentation and the community/devs are not able or willing to help on the Matrix chat or GitHub issues. I made an attempt, but I suspect I'm pretty much on a wrong path there.

I had more success with buildNpmPackage and then using your systemd.tmpfiles hack to inject that into node-red. But I couldn't figure out so far how to bundle more than one package this way. (Which appears to be easier with dream2nix.) Adding it to the node-red overlay with buildInputs = oldAttrs.buildInputs or [] ++ [ node-red-contrib-home-assistant-websocket ]; did not make any difference – which makes sense, since it's required during runtime. That's where your earlier suggestion would fit in, I assume.

Package example using buildNpmPackage

```nix { lib, buildNpmPackage, fetchFromGitHub }: buildNpmPackage rec { pname = "node-red-contrib-home-assistant-websocket"; version = "0.59.0"; src = fetchFromGitHub { owner = "zachowj"; repo = pname; rev = "v${version}"; hash = "sha256-BvptqazEfcRm5sdQrWNX46AfhLhPwAHC05fCHEOp9Xk="; }; npmDepsHash = "sha256-ShWmaBrZWruDiILaH2AGxtcuP2F9Sw8LXnHW31lnJVk="; # The prepack script runs the build script, which we'd rather do in the build phase. npmPackFlags = [ "--ignore-scripts" ]; meta = with lib; { description = "Node-RED integration with Home Assistant"; license = licenses.mit; }; } ```

In your example it appears you created a separate flake (and repo) that also contains the node package sources. What I'm after is a way to pull in a number of packages via GitHub (or npm) from my main NixOS configuration.

Would be great if you could shed some more light, this is slightly frustrating...

[fpetry]

I also stumbled upon the problem of not being able to install nodes from the palette manager.

The description of the option ´services.node-red.withNpmAndGcc´ says:

Give Node-RED access to NPM and GCC at runtime, so ‘Nodes’ can be downloaded and managed imperatively via the ‘Palette Manager’.

Even. If this is not the correct declarative way, I would expect it to work.

A way to declare the usage of arbitrary 3rd party nodes in a declarative way would be the best solution though. As a NixOS beginner, I did not understand the solution from @MatthewCroughan using dream2nix. Is there a comprehensive example somewhere?

This blocks moving my smarthome server to NixOS.

[MatthewCroughan]

I think the example I provided is comprehensive. You have to learn Nix, it's not automatic. @fpetry, though I think dream2nix/node2nix make it pretty easy.

[fpetry]

Ok. I ltried and learned and got it to work with several Node-Red modules separately. But when I'm trying to use multiple nodes at once, I fail at linking both to the module path.

Coming from your example:

  systemd.tmpfiles.rules = [
    "L+ ${config.services.node-red.userDir}/node_modules 0755 ${config.services.node-red.user} ${config.services.node-red.group} - ${node-red-dashboard}/lib/node_modules"
  ];

I wanted it to link to the subfolders like that:

  systemd.tmpfiles.rules = [
    "L+ ${config.services.node-red.userDir}/node_modules/node-red-dashboard 0755 ${config.services.node-red.user} ${config.services.node-red.group} - ${node-red-dashboard}/lib/node_modules/node-red-dashboard"
  ];

Which results in

Detected unsafe path transition /var/lib/node-red/node_modules (owned by node-red) → / (owned by root) during canonicalization of var/lib/node-red/node_modules.

So either my thought of linking is wrong here, or I need to change the owner of the module in the store somehow...

I'm aware, that this is not strictly related to this issue, but it can help to work around the problem and use Node-Red modules declaratively.

EDIT: I had to create the ${config.services.node-red.userDir}/node_modules/ folder manually. Now I'm looking into creating it with my configuration.