search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.52k stars 2.35k forks source link

nx migrate fails with Yarn berry #16338

Closed gremlin896 closed 8 months ago

gremlin896 commented 1 year ago

Current Behavior

Running nx migrate latest with yarn@3.5.0 fails.

Expected Behavior

nx migrate latest should run without any errors

GitHub Repo

https://github.com/nrwl/nx-examples

Steps to Reproduce

Run the following commands (assuming volta is installed)

  1. volta install yarn@3
  2. yarn
  3. nx migrate latest

Nx Report

>  NX   Report complete - copy this into the issue template

   Node : 16.17.0
   OS   : darwin arm64
   yarn : 3.5.0

   nx                      : 15.9.0-rc.2
   @nrwl/js                : 15.9.0-rc.2
   @nrwl/jest              : 15.9.0-rc.2
   @nrwl/linter            : 15.9.0-rc.2
   @nrwl/workspace         : 15.9.0-rc.2
   @nrwl/angular           : 15.9.0-rc.2
   @nrwl/cli               : 15.9.0-rc.2
   @nrwl/cypress           : 15.9.0-rc.2
   @nrwl/devkit            : 15.9.0-rc.2
   @nrwl/eslint-plugin-nx  : 15.9.0-rc.2
   @nrwl/react             : 15.9.0-rc.2
   @nrwl/tao               : 15.9.0-rc.2
   @nrwl/web               : 15.9.0-rc.2
   @nrwl/webpack           : 15.9.0-rc.2
   @nrwl/nx-cloud          : 15.3.3
   typescript              : 4.9.5
   ---------------------------------------
   Community plugins:
   @ngrx/component-store : 15.3.0
   @ngrx/effects         : 15.3.0
   @ngrx/entity          : 15.3.0
   @ngrx/router-store    : 15.3.0
   @ngrx/store           : 15.3.0
   @ngrx/store-devtools  : 15.3.0

Failure Logs

/bin/sh: /var/folders/tf/tjgc6tp54151mmr3jxmgcdr40000gq/T/tmp-32796-hV5BmtocRoSJ/node_modules/.bin/nx: No such file or directory
Command failed: /var/folders/tf/tjgc6tp54151mmr3jxmgcdr40000gq/T/tmp-32796-hV5BmtocRoSJ/node_modules/.bin/nx _migrate latest

Additional Information

After checking the temporary folders listed in the errors, there is no node_modules folder.

pumano commented 1 year ago

I use yarn 3.3.x and also 3.5.x and all works good

ChrisSargent commented 1 year ago

Also having the same issue. Tried doing it from v15 then manually upgraded to v16 and tried it, same result as above:

On yarn 3.5.1.

>  NX   Report complete - copy this into the issue template

   Node   : 18.16.0
   OS     : darwin x64
   yarn   : 3.5.1
   Hasher : Native

   nx                 : 16.2.1
   @nx/js             : 16.2.1
   @nx/jest           : 16.2.1
   @nx/linter         : 16.2.1
   @nx/workspace      : 16.2.1
   @nx/devkit         : 16.2.1
   @nx/eslint-plugin  : 16.2.1
   @nx/node           : 16.2.1
   @nx/react          : 16.2.1
   @nrwl/tao          : 16.2.1
   @nx/web            : 16.2.1
   @nx/webpack        : 16.2.1
   nx-cloud           : 16.0.5
   typescript         : 5.0.4
ciceropablo commented 1 year ago

I have the same issue. Any news?

 >  NX   Report complete - copy this into the issue template

   Node : 18.15.0
   OS   : darwin x64
   yarn : 3.5.1

   nx                      : 15.9.0
   @nrwl/js                : 15.9.0
   @nrwl/jest              : 15.9.0
   @nrwl/linter            : 15.9.0
   @nrwl/workspace         : 15.9.0
   @nrwl/cli               : 15.9.0
   @nrwl/cypress           : 15.9.0
   @nrwl/devkit            : 15.9.0
   @nrwl/eslint-plugin-nx  : 15.9.0
   @nrwl/react             : 15.9.0
   @nrwl/tao               : 15.9.0
   @nrwl/vite              : 15.9.0
   typescript              : 4.9.5
dlarroder commented 1 year ago

I have the same issue, any updates on this?

>  NX   Report complete - copy this into the issue template

   Node : 18.16.0
   OS   : darwin arm64
   yarn : 3.2.4

   nx : 15.6.3
   @nrwl/angular : Not Found
   @nrwl/cypress : 15.6.3
   @nrwl/detox : Not Found
   @nrwl/devkit : 15.6.3
   @nrwl/esbuild : Not Found
   @nrwl/eslint-plugin-nx : 15.6.3
   @nrwl/expo : Not Found
   @nrwl/express : 15.6.3
   @nrwl/jest : 15.6.3
   @nrwl/js : 15.6.3
   @nrwl/linter : 15.6.3
   @nrwl/nest : Not Found
   @nrwl/next : 15.6.3
   @nrwl/node : 15.6.3
   @nrwl/nx-cloud : 15.0.3
   @nrwl/nx-plugin : Not Found
   @nrwl/react : 15.6.3
   @nrwl/react-native : Not Found
   @nrwl/rollup : 15.6.3
   @nrwl/schematics : Not Found
   @nrwl/storybook : 15.6.3
   @nrwl/web : 15.6.3
   @nrwl/webpack : 15.6.3
   @nrwl/workspace : 15.6.3
   @nrwl/vite : Not Found
   typescript : 4.9.3
PatrickMennen commented 1 year ago

For me temporarily switching node versions from 16 to 20 fixed it

ajwootto commented 1 year ago

same issue trying to upgrade from nx 14 to nx 16, using yarn 3.6:


 >  NX   Report complete - copy this into the issue template

   Node : 18.16.0
   OS   : darwin arm64
   yarn : 3.6.0

   nx : 14.7.11
   @nrwl/angular : Not Found
   @nrwl/cypress : 16.3.2
   @nrwl/detox : 16.3.2
   @nrwl/devkit : 16.3.2
   @nrwl/eslint-plugin-nx : 16.3.2
   @nrwl/expo : Not Found
   @nrwl/express : Not Found
   @nrwl/jest : 16.3.2
   @nrwl/js : 16.3.2
   @nrwl/linter : 16.3.2
   @nrwl/nest : Not Found
   @nrwl/next : 16.3.2
   @nrwl/node : 16.3.2
   @nrwl/nx-cloud : Not Found
   @nrwl/nx-plugin : Not Found
   @nrwl/react : 16.3.2
   @nrwl/react-native : 16.3.2
   @nrwl/schematics : Not Found
   @nrwl/storybook : Not Found
   @nrwl/web : 16.3.2
   @nrwl/workspace : 16.3.2
   typescript : 5.0.4
   ---------------------------------------
   Local workspace plugins:
   ---------------------------------------
   Community plugins:
     @nrwl/webpack: 16.3.2
mpellerin42 commented 1 year ago

Same issue for me:

 >  NX   Report complete - copy this into the issue template

   Node : 18.15.0
   OS   : darwin arm64
   yarn : 3.3.1

   nx                      : 15.9.2
   @nrwl/js                : 15.9.2
   @nrwl/jest              : 15.9.2
   @nrwl/linter            : 15.9.2
   @nrwl/workspace         : 15.9.2
   @nrwl/angular           : 15.9.2
   @nrwl/cli               : 15.9.2
   @nrwl/cypress           : 15.9.2
   @nrwl/devkit            : 15.9.2
   @nrwl/eslint-plugin-nx  : 15.9.2
   @nrwl/node              : 15.9.2
   @nrwl/tao               : 15.9.2
   @nrwl/webpack           : 15.9.2
   typescript              : 4.9.5
   ---------------------------------------
   Community plugins:
   @nxext/vite   : 15.1.0
   @nxext/svelte : 15.7.0
   ng-mocks      : 14.8.0
   nx-electron   : 15.0.0-alpha.0
ChrisSargent commented 1 year ago

For me temporarily switching node versions from 16 to 20 fixed it

unfortunately not for me. Still failing with:

> % yarn nx migrate latest                 
/bin/sh: /var/folders/gr/94gk9mr11w90b8q6jt1701sm0000gn/T/tmp-96357-ymYkluNVHP0f/node_modules/.bin/nx: No such file or directory
Command failed: /var/folders/gr/94gk9mr11w90b8q6jt1701sm0000gn/T/tmp-96357-ymYkluNVHP0f/node_modules/.bin/nx _migrate latest

Would be really nice to get some input from the NX team on this... but then they also have more than 500 open issues :-(

Also get this after manually updating packages to @nx16...

ChrisSargent commented 1 year ago

Was able to get this to work by directly running ./node_modules/.bin/nx _migrate latest twice from the repo root. First pass seemed to update the packages, second pass created the migrations file.

igilham commented 1 year ago

I have the same issue but the workaround works: ./node_modules/.bin/nx _migrate latest

yarn nx report

 >  NX   Report complete - copy this into the issue template

   Node   : 18.12.1
   OS     : darwin arm64
   yarn   : 3.5.1
   Hasher : Native

   nx             : 16.3.2
   @nx/workspace  : 16.3.2
   @nx/devkit     : 16.3.2
   @nrwl/tao      : 16.3.2
   nx-cloud       : 16.0.5
   typescript     : 4.9.5
noamfeldman commented 1 year ago

Same issue happened to me as well when running the following. I was trying to upgrade from "nx": "15.7.0" to the latest:

> % nx migrate latest                 
/bin/sh: /var/folders/gr/{some temp directories path}/node_modules/.bin/nx: No such file or directory
Command failed: /var/folders/gr/{some temp directories path}/node_modules/.bin/nx _migrate latest

I took https://github.com/nrwl/nx/issues/16338#issuecomment-1588884718 as a workaround and it pass. Please note that the 2nd step is the actual migration: yarn nx migrate --run-migrations - it will fail as well with the same error above. Instead I run: yarn nx _migrate --run-migrations for this step as well and it passed successfully.

  NX   Report complete - copy this into the issue template

   Node   : 18.12.1
   OS     : darwin-arm64
   yarn   : 3.3.0

   nx                 : 16.4.2
   @nx/js             : 16.4.2
   @nx/jest           : 16.4.2
   @nx/linter         : 16.4.2
   @nx/workspace      : 16.4.2
   @nx/cypress        : 16.4.2
   @nx/devkit         : 16.4.2
   @nx/eslint-plugin  : 16.4.2
   @nx/react          : 16.4.2
   @nx/rollup         : 16.4.2
   @nrwl/tao          : 16.4.2
   @nx/web            : 16.4.2
   @nx/webpack        : 16.4.2
   nx-cloud           : 16.0.5
   typescript         : 5.1.6

Our project is a micro-frontend mono-repo based application. It has a container with main menu and user preferences, for the logged in user - this is the "host". Each page is developed by a deferent team and is a "remote" application. Before the upgrade we could share the host with the remote applications, for example - user selection of dark theme from the main menu affects all remote application (by using zustand as state management across the application). After the upgrade to NX 16 the state is not shared anymore by default as with version 15. To overcome it we changed the module-federation.config.js of the host application by adding the following:

  shared: (name, config) => {
    return { ...config, singleton: true };
  },
arvidboivie commented 1 year ago

There's another workaround that's setting YARN_NODE_LINKER="node-modules" for your command, i.e. YARN_NODE_LINKER="node-modules" yarn nx migrate latest.

I think the issue is that yarn defaults to pnp mode when used by nx, and it doesn't read your local .yarnrc file to check settings.

igilham commented 1 year ago

Interesting. I have the node linker mode set to node0-modules in my config but if it isn't being read then that won't help.

This suggests that the problem may be reproducible in Nx's automated testing in CI by running yarn with a few different configurations.

pongells commented 1 year ago

This fails with PNP enabled, as there is no node_modules/.bin/nx.

It is very annoying, but the "workaround" of falling back on node_modules via the YARN_NODE_LINKER="node-modules" environment variable seems to work..

YARN_NODE_LINKER="node-modules" yarn
YARN_NODE_LINKER="node-modules" yarn nx migrate latest
YARN_NODE_LINKER="node-modules" yarn nx migrate --run-migrations
yarn

(the last yarn just deletes all the node_modules that are not necessary because PNP)

ste7en commented 1 year ago

Still fails applying migrations with these workarounds. Yarn 3.4.1, Nx 16.7.2, Node 18.13.0.

Edit: Running YARN_NODE_LINKER="node-modules" NX_MIGRATE_SKIP_INSTALL=1 yarn nx migrate --run-migrations solved my issue as the error comes when Nx runs yarn install before running migrations

nemonemi commented 1 year ago

@pongells' solution worked for me, But this is far from ideal.

meeroslav commented 8 months ago

TL; DR; Workaround

For everyone affected by this problem, please follow the workaround proposed by @arvidboivie to use environment variable override. This is needed only for the nx migrate ... commands.

YARN_NODE_LINKER="node-modules" yarn nx migrate latest
yarn # no env var needed
YARN_NODE_LINKER="node-modules" yarn nx migrate --run-migrations

More details

When running nx migrate ... commands to upgrade from version A to version B, we need to run the migration command using the new version B. In order to do this, we generate a temp folder where we install version B and then run the nx migrate pointing to your repo. This works as expected for all cases apart from one. Sadly, that one affects you.

If you use Yarn Berry with nodeLinker set to node-modules everything works fine inside your repo, but that temp folder doesn't know about the nodeLinker settings (which is a bug we need to fix) so it uses the default, PnP installation.

Based on your repo, nx is still expecting to find binary in the node_modules\nx\.bin, but that doesn't exist for PnP, and you end up with that ugly error.

Solution

Now that we know the problem, we'll see the fix is implemented as soon as possible. Sorry for the delay.

Thank you all for your contribution to clarification of this issue.

github-actions[bot] commented 7 months ago

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.