yarn sets $NODE env variable to the node version from nix which breaks some usages

[ahmedelgabri]

Describe the bug

nixpkgs.yarn (or nixpkgs.nodePackages.yarn) sets $NODE to a specific node version, which will it error out when used in a project that has strict engine requirements inside package.json

Steps To Reproduce

I created a simple repo to be able to reproduce this issue https://github.com/ahmedelgabri/nix-yarn-issue

1. make sure to install nixpkgs.yarn & nixpkgs.fnm (or nvm)
2. Clone the repo
3. Run fnm use --install-if-missing to install the version from .nvmrc file
4. Validate that you get the correct version node -v & which node should point to the node coming from fnm
5. Run yarn start, it shouldn't run & you will get this error

error nix-yarn-issue@1.0.0: The engine "node" is incompatible with this module. Expected version ">= 16". Got "14.18.1"
error Commands cannot run with an incompatible environment.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

6. Run this command

yarn env | grep -E "\"(NODE|npm_node_execpath|npm_config_user_agent)\""

The output should look like this

"NODE": "/nix/store/d37aprv8pxqpslag1yq6n0q2q0fnpqwg-nodejs-14.18.1/bin/node",
"npm_node_execpath": "/nix/store/d37aprv8pxqpslag1yq6n0q2q0fnpqwg-nodejs-14.18.1/bin/node",
"npm_config_user_agent": "yarn/1.22.17 npm/? node/v14.18.1 darwin x64",

The problem is that nix sets the NODE environment variable to the nix node executable, which then yarn picks up and sets npm_node_execpath to the same value.

I tried to search the repo to find where exactly this is happening but I couldn't really find it.

Overriding NODE is not the solution because this doesn't affect npm_config_user_agent & will still error out.

So doing this doesn't work

export NODE=$(which node) yarn start

It will result in the same error here

error nix-yarn-issue@1.0.0: The engine "node" is incompatible with this module. Expected version ">= 16". Got "14.18.1"
error Commands cannot run with an incompatible environment.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

because the output looks like this

"NODE": "/Users/ahmed/.fnm/node-versions/v16.13.0/installation/bin/node",
"npm_node_execpath": "/Users/ahmed/.fnm/node-versions/v16.13.0/installation/bin/node",
"npm_config_user_agent": "yarn/1.22.17 npm/? node/v14.18.1 darwin x64", # this is not correct still

This problem doesn't happen if yarn is installed using homebrew or directly curl -o- -L https://yarnpkg.com/install.sh | bash.

This is the result when using yarn which is installed using homebrew

"NODE": "/Users/ahmed/.fnm/node-versions/v16.13.0/installation/bin/node",
"npm_node_execpath": "/Users/ahmed/.fnm/node-versions/v16.13.0/installation/bin/node",
"npm_config_user_agent": "yarn/1.22.17 npm/? node/v16.13.0 darwin x64",

Expected behavior

That yarn works correctly & switch NODE, npm_node_execpath & npm_config_user_agent when the node version is changed.

Additional context

Add any other context about the problem here.

Notify maintainers

@DamienCassou (sorry if you are not the maintainer but I couldn't mention anyone from the maintainer list)

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-darwin"`
 - host os: `Darwin 21.1.0, macOS 10.16`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.4pre20210326_dd77f71`
 - channels(ahmed): `"darwin, nixpkgs-21.05pre286546.2cca79be09c"`
 - nixpkgs: `/nix/store/vpgzs19895khmlgjryi4y9w7cd2br1m5-source`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[DamienCassou]

I have no clue about this issue, I'm sorry.

/cc @offlinehacker @screendriver

[Artturin]

nix why-depends ".#yarn" ".#nodejs" --all
/nix/store/yg5c88hv8p37rd1nm2s7ly6hp8jq9byh-yarn-1.22.17
└───libexec/yarn/bin/yarn.js: …#!/nix/store/rdagrwgmq8n9jcs5iq5xpcxcxnh0lcc7-nodejs-14.18.1/bin/node../* esl…
    libexec/yarn/bin/yarnpkg: …#!/nix/store/rdagrwgmq8n9jcs5iq5xpcxcxnh0lcc7-nodejs-14.18.1/bin/node.require…
    libexec/yarn/lib/cli.js: …#!/nix/store/rdagrwgmq8n9jcs5iq5xpcxcxnh0lcc7-nodejs-14.18.1/bin/node.module.…
    → /nix/store/rdagrwgmq8n9jcs5iq5xpcxcxnh0lcc7-nodejs-14.18.1

there's a nodejs14 shebang in those so this might be the reason

#!/nix/store/rdagrwgmq8n9jcs5iq5xpcxcxnh0lcc7-nodejs-14.18.1/bin/node

[ahmedelgabri]

@Artturin any clue how can this be changed? looking at the derivation it seems like something happens at the build phase because the install phase is a simple symlinking https://github.com/NixOS/nixpkgs/blob/5715d5b507362b1d9c87317d874e47e33b0eabd6/pkgs/development/tools/yarn/default.nix

[samuela]

I'm seeing the exact same issue using the nodejs-16_x and yarn packages as of c11d08f02390aab49e7c22e6d0ea9b176394d961. I have the following in my package.json:

  "engines": {
    "node": ">=16"
  },

But I'm seeing the following when trying to run yarn:

[nix-shell:~/dev/cuddlefish/next]$ node --version
v16.13.0

[nix-shell:~/dev/cuddlefish/next]$ yarn run dev
yarn run v1.22.17
error next@: The engine "node" is incompatible with this module. Expected version ">=16". Got "14.18.1"
error Commands cannot run with an incompatible environment.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

which is obviously a contradiction. I have no idea where 14.18.1 is coming from besides the fact that it is the current nodejs version for the nixpkgs commit I'm on.

FWIW:

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.1.0, macOS 12.0.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.4`
 - channels(samuelainsworth): `"darwin, home-manager, nixpkgs-21.11pre330734.5cb226a06c4"`
 - channels(root): `"nixpkgs-21.11pre330734.5cb226a06c4"`
 - nixpkgs: `/Users/samuelainsworth/.nix-defexpr/channels/nixpkgs`

[samuela]

Any known workaround for this?

EDIT: Using an override works for me:

    (yarn.override {
      nodejs = nodejs-16_x;
    })

[ahmedelgabri]

@samuela override doesn't solve the problem if you work with different projects & everyone requires a different node version this will still break.

[samuela]

Totally, it's just a hack for the time being. I'd still like to see this resolved properly as well!

[thall]

With overlay:

nixpkgs.overlays = [(
  self: super: {
    yarn = super.yarn.override {
      nodejs = pkgs.nodejs-16_x;
    };
  }
)];

[ahmedelgabri]

@thall This doesn't solve the problem. Because now yarn will not work with any project that's using an older version

https://github.com/NixOS/nixpkgs/issues/145634#issuecomment-973978780

The reason why the shebang should stay the same is this, yarn will work as expected in any project & picking up the project version correctly.

[spencerpogo]

The override works for pkgs.yarn but not pkgs.nodePackages.yarn.

[thall]

@thall This doesn't solve the problem. Because now yarn will not work with any project that's using an older version

#145634 (comment)

@ahmedelgabri Yes, I know that, its not a solution, just a workaround that I wanted to share.

[j-a-m-l]

I've also this problem, @ahmedelgabri, and I've fixed it in a shell.nix to have a reproducible local environment for each project:

New project:

{ pkgs ? import <nixpkgs> {
  overlays = [(
    self: super: {
      yarn = super.yarn.override {
        nodejs = pkgs.nodejs-18_x;
      };
    }
  )];
} }:
  pkgs.mkShell {
    nativeBuildInputs = with pkgs; [
      nodejs-18_x
      yarn
    ];
}

Other, older, project, does not need the overlay:

{ pkgs ? import <nixpkgs> { } }:
  pkgs.mkShell {
    nativeBuildInputs = with pkgs; [
      nodejs-16_x
      yarn
    ];
}

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-use-pnpm-with-recent-nodejs/21867/1

[tennox]

The override works for pkgs.yarn but not pkgs.nodePackages.yarn.

To resolve this (also useful for e.g. nodePackages.pnpm) it can be achieved by overlaying the default nodejs package:

overlays = [ (self: prevPkgs: {
    nodejs = prevPkgs.nodejs-16_x;
}) ]

which is then used to build nodePackages, so nixpkgs.nodePackages.pnpm will be using that version of nodejs:

$ pnpm node --version
v16.17.1

[kagworld]

seems to be using the wrong version

$ yarn node -v
yarn node v1.22.18
v14.20.1
...

workaround

$ node `which yarn` node -v
yarn node v1.22.18
v18.9.1

nodePackages.yarn pinned the nodejs version that got installed the first time

$ head -1 `which yarn`
#!/nix/store/4fxksc2amz0s8cb0hxnwrcnrc17zdm70-nodejs-14.20.1/bin/node

[davidpfarrell]

I was in the process of creating a new issue but luckily discovered this so I'll share my thoughts/findings here:

---

The decision to symlink yarn to bin/yarn.js instead of bin/yarn makes it impossible invoke yarn directly using a custom node version.

The official yarn package has yarn as a shell script and yarn.js as a node script

bin/yarn

#!/bin/sh

bin/yarn.js

#!/usr/bin/env node

Additionally, yarn.js will use whichever node version is first on the path.

HOWEVER, after being nixified, the #! expressions become hard-coded and we lose the dynamic nature of the original:

fresh install of yarn

$ nix-shell -p yarn

$ which yarn
/nix/store/xblz3z89rsnfmmj1zyji3n3v2rvrp2m4-yarn-1.22.19/bin/yarn

check path-installed commands

$ ls -l `dirname \`which yarn\``

total 0
lrwxr-xr-x 1 root nixbld 27 Dec 31  1969 yarn -> ../libexec/yarn/bin/yarn.js
lrwxr-xr-x 1 root nixbld 27 Dec 31  1969 yarnpkg -> ../libexec/yarn/bin/yarn.js

check #! settings of installed commands

$ head -1 `dirname \`realpath \\\`which yarn\\\`\``/yarn{,.js}

==> /nix/store/xblz3z89rsnfmmj1zyji3n3v2rvrp2m4-yarn-1.22.19/libexec/yarn/bin/yarn <==
#!/nix/store/ka6rabx4lz7m3habrjhh8hvbgxbz8r98-bash-5.2-p15/bin/sh

==> /nix/store/xblz3z89rsnfmmj1zyji3n3v2rvrp2m4-yarn-1.22.19/libexec/yarn/bin/yarn.js <==
#!/nix/store/kvlngvm6agicknyj02m93zbql3g39kw2-nodejs-18.15.0/bin/node

So when yarn.js is invoked directly, it will ALWAYS use whichever version of node is hard-coded in the #!.

However, if we could invoke the yarn shell script, it will check the path for which node version to use.

My recommended fix:

Modify the nix pkg installation steps to symlink yarn to the yarn shell script and not the yarn.js node script.

Additionally, I think hard-coding the #! in the yarn.js script is a bit of a bug, but not sure if there's a configuration option in the nixpkg to skip the fixup for that script?