search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.25k stars 2.31k forks source link

External dependencies missing in projectGraph on Windows #19520

Open JonWallsten opened 11 months ago

JonWallsten commented 11 months ago

Current Behavior

I'm currently investigating why module federation is broken in nx >= 16.8.1, https://github.com/nrwl/nx/issues/19121. The root cause is that externalModules are missing in the projectGraph, so host/remote cannot share dependencies.

After a lot of digging it seems like the reason is that the WorkspaceContext is missing externalModules. In nx@16.2.2 the external modules are added to the project graph by calling buildNpmPackageNodes. In nx@16.8.1 something called getNxWorkspaceFilesFromContext is used instead. It's using a WorkspaceContext to retrieve the externalModules, but it's always empty.

Since WorkspaceContext is created by a binary I'm not sure if I can investigate this further. But since this is only an issue on Windows and this is a native module I guess that it's not that far fetched that it could be the source of the issue. 

projectGraph {
  nodes: {
    remote: { name: 'remote', type: 'app', data: [Object] },
    host: { name: 'host', type: 'app', data: [Object] }
  },
  externalNodes: {},
  dependencies: { remote: [], host: [] },
  version: '6.0'
}

Expected Behavior

WorkspaceContext should add npm dependencies as externalModules so it can be added to the projectGraph

GitHub Repo

https://github.com/JonWallsten/nx-repro/tree/dynamic

Steps to Reproduce

  1. npm i
  2. npm run build

Nx Report

Node   : 18.14.1
   OS     : win32-x64
   npm    : 9.3.1

   nx                 : 16.10.0
   @nx/js             : 16.10.0
   @nx/jest           : 16.10.0
   @nx/linter         : 16.10.0
   @nx/workspace      : 16.10.0
   @nx/angular        : 16.10.0
   @nx/cypress        : 16.10.0
   @nx/devkit         : 16.10.0
   @nx/eslint-plugin  : 16.10.0
   @nrwl/tao          : 16.10.0
   @nx/web            : 16.10.0
   @nx/webpack        : 16.10.0
   typescript         : 5.1.6

Failure Logs

No response

Package Manager Version

No response

Operating System

Additional Information

No response

JonWallsten commented 10 months ago

@AgentEnder Is there anything I can do to help with this?

JonWallsten commented 9 months ago

Just updated to nx@17.x.x and this is still an issue. So we're still stuck on 16.2.2. Is there any plan to make this work on Windows? I'm already offered to help, but I need support to get further. Can anyone else assist me in this? @FrozenPandaz / @juristr / @Cammisuli / @Coly010

JonWallsten commented 9 months ago

Finally figured out how to use --inspect-brk on the child process that creates the graph. So now I at least have a chance to figure this out: image I'm trying to find out why the externalNodes is always empty. I think that's the key. Any help appreciated.

JonWallsten commented 9 months ago

Okey, I finally found the issue! 🥳 I tried to find all the places where externalNodes are affected somehow. And found this getNodes function in npm-parser.ts (we're using npm). It takes data: NpmLockFile as input. But I'm not using a lock-file. So added package-lock=true to the .npmrc and built again, but it still didn't work. I remember seeing .gitignore somewhere when I debugged. So I removed package-lock.json from .gitignore and finally it worked.

But since I now have to push the package-lock.json to our repo, because it's no longer in the .gitignore file I can't get the build to work in Gitlab, because the lock file was prepared on windows.

I get this error: Error: Cannot find module '@nx/nx-linux-x64-gnu'

So I guess we have two issue here:

  1. It should work without a package-lock.json
  2. package-lock.json should not be ignored when building the graph But if 1 is fixed, 2 is not necessary to fix.

@AgentEnder

Cammisuli commented 9 months ago

Make sure the lock file is regenerated. Follow the instructions here: https://nx.dev/recipes/troubleshooting/troubleshoot-nx-install-issues#how-to-fix

Basically the lock file should include all the optional platform deps, so that the single lock file can be used on multiple systems.

Otherwise, if you really want to keep the lock file in the gitignore, you could try adding it to the .nxignore file with !package-lock.json

JonWallsten commented 9 months ago

Make sure the lock file is regenerated. Follow the instructions here: https://nx.dev/recipes/troubleshooting/troubleshoot-nx-install-issues#how-to-fix

Basically the lock file should include all the optional platform deps, so that the single lock file can be used on multiple systems.

Otherwise, if you really want to keep the lock file in the gitignore, you could try adding it to the .nxignore file with !package-lock.json

We're using npm ci --cache .npm --prefer-offline to install. And i can see this in the package-lock.json: image

But I still get the same error. I've tried to clear the runner's cache as well.

Cammisuli commented 9 months ago

on the CI, can you try running find node_modules -name "*.node" | grep -E "(@nx|@nrwl)" and making sure that the linux binary shows up there?

For example, on my computer running the command on the nx repo, i get this result:

node_modules/.pnpm/@nrwl+nx-darwin-arm64@15.8.0/node_modules/@nrwl/nx-darwin-arm64/nx.darwin-arm64.node
node_modules/.pnpm/@nx+nx-darwin-arm64@17.1.1/node_modules/@nx/nx-darwin-arm64/nx.darwin-arm64.node
JonWallsten commented 9 months ago

@Cammisuli: Not sure what going on, I just won't work with the package-lock.json. I've tried all that I can think of. Removing the package-lock file is the only thing I can get to work. But then the issue is that I don't have a package-lock file when building. So I end up with the same problem as locally and the loop is closed! 

npm install || true
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules/esbuild',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/esbuild/bin'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/esbuild/bin'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules/@swc',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/@swc'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/@swc'
npm WARN cleanup     }
npm WARN cleanup   ],
npm WARN cleanup   [
npm WARN cleanup     '/builds/....../node_modules',
npm WARN cleanup     [Error: ENOTEMPTY: directory not empty, rmdir '/builds/....../node_modules/@swc'] {
npm WARN cleanup       errno: -39,
npm WARN cleanup       code: 'ENOTEMPTY',
npm WARN cleanup       syscall: 'rmdir',
npm WARN cleanup       path: '/builds/....../node_modules/@swc'
npm WARN cleanup     }
npm WARN cleanup   ]
npm WARN cleanup ]
npm ERR! code 1
npm ERR! path /builds/....../node_modules/nx
npm ERR! command failed
npm ERR! command sh -c node ./bin/post-install
npm ERR! /builds/....../node_modules/nx/src/native/index.js:244
npm ERR!     throw loadError
npm ERR!     ^
npm ERR! 
npm ERR! Error: Cannot find module '@nx/nx-linux-x64-gnu'
npm ERR! Require stack:
npm ERR! - /builds/....../node_modules/nx/src/native/index.js
npm ERR! - /builds/....../node_modules/nx/src/hasher/task-hasher.js
npm ERR! - /builds/....../node_modules/nx/src/tasks-runner/run-command.js
npm ERR! - /builds/....../node_modules/nx/src/nx-cloud/utilities/get-cloud-options.js
npm ERR! - /builds/....../node_modules/nx/bin/post-install.js
npm ERR!     at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
npm ERR!     at Module._load (node:internal/modules/cjs/loader:922:27)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:11[43](https://gitlab.scania.com/....../-/jobs/32749823#L43):19)
npm ERR!     at require (node:internal/modules/cjs/helpers:119:18)
npm ERR!     at Object.<anonymous> (/builds/....../node_modules/nx/src/native/index.js:184:31)
npm ERR!     at Module._compile (node:internal/modules/cjs/loader:12[56](https://gitlab.scania.com/....../-/jobs/32749823#L56):14)
npm ERR!     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
npm ERR!     at Module.load (node:internal/modules/cjs/loader:1119:32)
npm ERR!     at Module._load (node:internal/modules/cjs/loader:9[60](https://gitlab.scania.com/....../-/jobs/32749823#L60):12)
npm ERR!     at Module.require (node:internal/modules/cjs/loader:1143:19) {
npm ERR!   code: 'MODULE_NOT_FOUND',
npm ERR!   requireStack: [
npm ERR!     '/builds/....../node_modules/nx/src/native/index.js',
npm ERR!     '/builds/....../node_modules/nx/src/hasher/task-hasher.js',
npm ERR!     '/builds/....../node_modules/nx/src/tasks-runner/run-command.js',
npm ERR!     '/builds/....../node_modules/nx/src/nx-cloud/utilities/get-cloud-options.js',
npm ERR!     '/builds/....../node_modules/nx/bin/post-install.js'
npm ERR!   ]
npm ERR! }
npm ERR! 
npm ERR! Node.js v18.18.2
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-12-04T07_03_29_[79](https://gitlab.xxxxxxxxx.com/....../-/jobs/32749823#L79)6Z-debug-0.log
$ find node_modules -name "*.node" | grep -E "(@nx|@nrwl)"
Cleaning up project directory and file based variables

Installing it manually works:

$ npm install @nx/nx-linux-x64-gnu || true
added 1930 packages, and audited 1931 packages in 15s
270 packages are looking for funding
  run `npm fund` for details
5 moderate severity vulnerabilities
To address issues that do not require attention, run:
  npm audit fix
To address all issues (including breaking changes), run:
  npm audit fix --force
Run `npm audit` for details.
$ find node_modules -name "*.node" | grep -E "(@nx|@nrwl)"
node_modules/@nx/nx-linux-x64-gnu/nx.linux-x64-gnu.node
Cleaning up project directory and file based variables
00:00
Job succeeded

It only worked until I tried to build:

>  NX   Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.

This ended up working (but is of course a bad solution):

 - npm i @rollup/rollup-linux-x64-gnu @nx/nx-linux-x64-gnu --cache .npm --prefer-offline

We need to use the package-lock.json file since we use Gitlabs cache feature for the npm modules. So we can't rely on creating the package-lock.json at install, since the command is nog run when cache is used.

Cammisuli commented 9 months ago

Are the optional dependencies listed in the package-lock.json file? The fact that you can't find rollup's native binary either is suspicious of the package-lock. All the optional dependencies should be listed in the package-lock. For example this is how the nx ones look:

    "node_modules/@nx/nx-darwin-arm64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-darwin-arm64/-/nx-darwin-arm64-17.2.2.tgz",
      "integrity": "sha512-8dPPmX8KQT/HtizaDmz/+TIAytAwuC0Mvx/UqhiFNL5l0o5HnK+AfNEe2GWk9kfZjg22nmMfBIBDIxl6odNVBQ==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "darwin"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-darwin-x64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-darwin-x64/-/nx-darwin-x64-17.2.2.tgz",
      "integrity": "sha512-hJaqyF+wAdFznXv2WQ+qTkCX5+Dr4cCtA9M67/Rln70bUkiQ0upQAacI0BUrHLZOKAS6OKW2V37V1xgCLk+NHw==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "darwin"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-freebsd-x64": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-freebsd-x64/-/nx-freebsd-x64-17.2.2.tgz",
      "integrity": "sha512-CxbtqfL33MHqaEXKNjgrZCf9Sot13KmZvkbET/DOQGJVX6yD1kKV8yTN9KfoeUwf1UorCCyTy5NdyFKq0nkIOQ==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "freebsd"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm-gnueabihf": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm-gnueabihf/-/nx-linux-arm-gnueabihf-17.2.2.tgz",
      "integrity": "sha512-r5kAqC3c6kIXUfDK27zQ3LsZO/tsDzmiNilxc00DmUIzLPBLt3qlXbctD5jxE80xVkgNSCEZtSFIgTpYdymnAQ==",
      "cpu": [
        "arm"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm64-gnu": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm64-gnu/-/nx-linux-arm64-gnu-17.2.2.tgz",
      "integrity": "sha512-EowQSciBfUg82jwHruFhCnY6/raWoo8FsGPaTNaKPcXtckYCi1hj9XUgHpeG06Z5lgPvO/1pFIUSSkIc3vefQA==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-arm64-musl": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-arm64-musl/-/nx-linux-arm64-musl-17.2.2.tgz",
      "integrity": "sha512-MQgXf5u12EyCINLiuntjOE5sR2Vp4CjHMrXt8aT8ZRURUc6PBkubDjslVhVJ7Xmot38i5Fh/ngrdXzFrQETAUw==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-x64-gnu": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-x64-gnu/-/nx-linux-x64-gnu-17.2.2.tgz",
      "integrity": "sha512-qPB42G0xLlwbpC40r8RxgS+xsKoURpkBmcwhXSo+5iBnfOvW2qk4ox+0h+l+mTT8RB30H/7XNEEXergu2Px1Zw==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-linux-x64-musl": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-linux-x64-musl/-/nx-linux-x64-musl-17.2.2.tgz",
      "integrity": "sha512-AnD7rtxyU6LIF0ewCO0gmvrQuJPSPZPXAzrUn95YP+nh357bfweOPV6uyHYn/Nqax1f3FPFe7Ervo48TAzunug==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "linux"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-win32-arm64-msvc": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-win32-arm64-msvc/-/nx-win32-arm64-msvc-17.2.2.tgz",
      "integrity": "sha512-xNvdlfGnTBx6FygCR+AenduVWATh4HtaNibFM7ELdsTM4posYFYk7jpNXf5dFtSANJ+Is2vWAEqjWx1Th0/ZjQ==",
      "cpu": [
        "arm64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "win32"
      ],
      "engines": {
        "node": ">= 10"
      }
    },
    "node_modules/@nx/nx-win32-x64-msvc": {
      "version": "17.2.2",
      "resolved": "http://localhost:4876/@nx%2fnx-win32-x64-msvc/-/nx-win32-x64-msvc-17.2.2.tgz",
      "integrity": "sha512-2pULDtVSGJUYiFKFJTASaOVpSF9AkjOvVgkRTV92Cx2ykT/9wwg7EfY/WZmLxfAkYFjiQyD9s4Wq06yp/rgd4w==",
      "cpu": [
        "x64"
      ],
      "dev": true,
      "optional": true,
      "os": [
        "win32"
      ],
      "engines": {
        "node": ">= 10"
      }
    },

Other package's optional native binaries should be around the same area too.

Did you try deleting node_modules + the package-lock and doing npm i again? Did you try updating your npm version?

sagarbasavaraj commented 6 months ago

I am having the same problem with npm but it worked with yarn package manager as we have yarn.lock file.

gabynevada commented 6 months ago

I switched our package manager to bun and having the same issue as it uses the bun/lockb file.

matheo commented 5 months ago

I'm using pnpm and I added pnpm-lock.yaml to .nxignore to prevent nx format issues but then my dep-graph was messed up without external dependencies :thinking: