Nx 19.5 cannot find projects after initial creation

SiKreuz commented 1 month ago

Current Behavior

I created a new project with nx g @nx/nest:app backend. Process went through without any errors. However, If I now try to compile, nx tells me it cannot find the projects. Calling nx show projects gives me an empty list.

This happens from version 19.5.0 on. I downgraded to 19.4.4 and there it does work. nx show projects gives me all existing projects. Upgrading back to 19.5.0 lets the projects vanish again

Expected Behavior

I expect a freshly generated project to appear and be found when using nx show projects.

GitHub Repo

No response

Steps to Reproduce

bunx create-nx-workspace my-workspace
cd my-workspace
bun nx add @nx/nest
bun nx g @nx/nest:app my-nest-app
bun nx show projects

Nx Report

Node           : 22.5.1
OS             : darwin-arm64
Native Target  : aarch64-macos
bun            : 1.1.22

nx                 : 19.5.7
@nx/js             : 19.5.7
@nx/jest           : 19.5.7
@nx/linter         : 19.5.7
@nx/eslint         : 19.5.7
@nx/workspace      : 19.5.7
@nx/devkit         : 19.5.7
@nx/eslint-plugin  : 19.5.7
@nx/nest           : 19.5.7
@nx/node           : 19.5.7
@nrwl/tao          : 19.5.7
@nx/web            : 19.5.7
@nx/webpack        : 19.5.7
typescript         : 5.4.5
---------------------------------------
Registered Plugins:
@nx/webpack/plugin
@nx/eslint/plugin
@nx/jest/plugin

Failure Logs

No response

Package Manager Version

bun 1.1.22

Operating System

[X] macOS
[ ] Linux
[ ] Windows
[ ] Other (Please specify)

Additional Information

No response

pascalmann commented 1 month ago

Same problem for me (Ubuntu 20). Some libs disappear from the project graph. Nx reset doesn't solve the problem.

If I save the project.json file of a lib
nx show projects -> the lib returns in the project graph
nx reset -> the lib disappear again

mechanical-fish commented 3 weeks ago

This is probably the bug that just bit me (on MacOS Sonoma 14.6.1)-- because reverting to 19.4.4 also brought back my project graph, whereas 19.5.7 and 19.6.1 just return an empty project graph.

jaysoo commented 3 weeks ago

I'm not able to reproduce on either 19.5 or 19.6. Can someone link a repro? I'm also on MacOS Sonoma 14.6.1.

Does running with NX_DAEMON=false NX_CACHE_PROJECT_GRAPH=false generate the graph?

mechanical-fish commented 3 weeks ago

Running with NX_DAEMON=false NX_CACHE_PROJECT_GRAPH=false does not help with Nx 19.5.7. npx nx show projects still returns nothing.

I've just managed to reproduce this on a fresh clone of https://github.com/nrwl/nx-examples with Nx upgraded to 19.6.1. After doing so, npx nx show projects on that repo returns nothing, and NX_DAEMON=false npx nx show-projects hangs indefinitely and has to be terminated with kill -9, leaving behind a whole bunch of plugin-worker processes. Downgrading to 19.4.4 resolves the issue and gives me a list of projects.

I'll try running something like @SiKreuz 's workflow to see if I can find an even more minimal example.

I note that while we have many programmers running laptops set up much like mine, and who have run Nx 19.5.7 on existing working copies of our codebase without a hitch, I'm the first and only one to report this problem. I wish I could guess what I did differently, and why that difference persists on my machine through a reboot, a cleanup of my $HOME directory, a freshly cloned and installed codebase, an entirely new Nx project, etc. (We've now reverted our Nx-equipped codebase to 19.4.4, so this problem probably won't bite anyone else around here.)

mechanical-fish commented 3 weeks ago

EDIT: I can't get `create-nx-workspace` to work without hanging even back in nx version `18.3.4`, so 🤷

npx create-nx-workspace my-workspace also hangs indefinitely for me with version 19.6.1:

❯ npx create-nx-workspace my-workspace

 NX   Let's create a new workspace [https://nx.dev/getting-started/intro]

✔ Which stack do you want to use? · none
✔ Package-based monorepo, integrated monorepo, or standalone project? · package-based
✔ Which CI provider would you like to use? · skip
✔ Would you like remote caching to make your build faster? · skip

 NX   Creating your v19.6.1 workspace.

✔ Installing dependencies with npm
⠦ Creating your workspace in my-workspace

I can stop the install process with ^C but the process list continues to contain three plugin-worker processes, plus these two processes which need to be killed with kill -9:

node nx new --interactive --standaloneApi --routing --no-skipGit --no-g --commit.message=Initial commit --no-allPrompts --no-a --/bin/sh=/Users/mbooth/.npm/_npx/d2207cf76adb22dc/node_modules/.bin/create-nx-workspace --name=my-workspace --stack=none --preset=npm --nxCloud=skip --packageManager=npm --defaultBase=main --nxWorkspaceRoot=/Users/mbooth/Code/vendor/reprobug

npm exec nx new --interactive --standaloneApi --routing --no-skipGit --no-g --commit.message=Initial commit --no-allPrompts --no-a --/bin/sh=/Users/mbooth/.npm/_npx/d2207cf76adb22dc/node_modules/.bin/create-nx-workspace --name=my-workspace --stack=none --preset=npm --nxCloud=skip --packageManager=npm --defaultBase=main --nxWorkspaceRoot=/Users/mbooth/Code/vendor/reprobug

mechanical-fish commented 3 weeks ago

we are now at the stage of debugging where I doubt the nature of reality. In a directory with only one file, package.json, with the following contents:

{
  "name": "my-workspace",
  "version": "0.0.0",
  "license": "MIT",
  "scripts": {},
  "private": true,
  "dependencies": {},
  "devDependencies": {
    "nx": "17"
  }
}

... npm install && npx nx init installs Nx version 17.3.2 and succeeds with exit code 0.

Changing the specification to "nx": "18" installs Nx version 18.3.5 but npx nx init prints Checking dependencies and then hangs, apparently forever. I can use Ctrl-C to kill that process.

Changing the specification to "nx": "19" installs Nx version 19.6.1 and npx nx init prints:

npx nx init      

 NX   🐳 Nx initialization

 NX   📦 Installing dependencies

up to date, audited 115 packages in 290ms

22 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
✔ Would you like remote caching to make your build faster? · skip

 NX   🧐 Checking dependencies

... and then hangs forever. This cannot be cancelled with Ctrl-C, I have to run killall -9 node to stop it.

I'm afraid I don't know anymore if I'm reporting the same bug or a completely different bug, but perhaps you'll know more about what to make of this than I do.

jaysoo commented 3 weeks ago

@mechanical-fish Are there logs from .nx/cache/d/daemon.log?

jaysoo commented 3 weeks ago

@SiKreuz @pascalmann Can you try with NX_DAEMON=false and see if that helps? Also try NX_CACHE_PROJECT_GRAPH=false if turning daemon off didn't help.

SiKreuz commented 3 weeks ago

@jaysoo I'm currently on vacation. I can check and reply again in one or two weeks latest.

mechanical-fish commented 3 weeks ago

After reinstalling node_modules for my project and clearing the .nx directory, I ran npx nx show projects (still prints nothing) and the .nx/workspace-data/d/daemon.log looks like:

[NX Daemon Server] - 2024-08-22T23:53:21.222Z - Started listening on: /var/folders/rk/q3h_l_6941n3lgs25bmmk6m00000gp/T/949b80dc448f69915e41/d.sock
[NX Daemon Server] - 2024-08-22T23:53:21.224Z - [WATCHER]: Subscribed to changes within: /Users/mbooth/Code/project (native)
[NX Daemon Server] - 2024-08-22T23:53:21.230Z - Established a connection. Number of open connections: 1
[NX Daemon Server] - 2024-08-22T23:53:21.231Z - Established a connection. Number of open connections: 2
[NX Daemon Server] - 2024-08-22T23:53:21.231Z - Closed a connection. Number of open connections: 1
[NX Daemon Server] - 2024-08-22T23:53:21.232Z - [REQUEST]: Client Request for Project Graph Received
[NX Daemon Server] - 2024-08-22T23:53:21.323Z - [REQUEST]: Updated workspace context based on watched changes, recomputing project graph...
[NX Daemon Server] - 2024-08-22T23:53:21.324Z - [REQUEST]: 
[NX Daemon Server] - 2024-08-22T23:53:21.324Z - [REQUEST]: 
[NX Daemon Server] - 2024-08-22T23:53:21.324Z - Time taken for 'loadNxPlugins' 90.6717080026865ms
[NX Daemon Server] - 2024-08-22T23:53:21.328Z - Time taken for 'build-project-configs' 1.1758749932050705ms
[NX Daemon Server] - 2024-08-22T23:53:21.330Z - [REQUEST]: Responding to the client. project-graph
[NX Daemon Server] - 2024-08-22T23:53:21.330Z - Done responding to the client project-graph
[NX Daemon Server] - 2024-08-22T23:53:21.331Z - Handled REQUEST_PROJECT_GRAPH. Handling time: 98. Response time: 1.
[NX Daemon Server] - 2024-08-22T23:53:21.331Z - Time taken for 'total for creating and serializing project graph' 98.11312499642372ms
[NX Daemon Server] - 2024-08-22T23:53:21.353Z - Closed a connection. Number of open connections: 0

There is a daemon process running now on my machine, plus three nx/src/project-graph/plugins/isolation/plugin-worker processes.

NX_DAEMON=false NX_CACHE_PROJECT_GRAPH=false npx nx show projects --verbose still prints nothing.

pascalmann commented 3 weeks ago

@jaysoo ~I no longer have the problem with this version "nx": "19.6.2"~.

My problem was elsewhere. See below.

mechanical-fish commented 3 weeks ago

Sadly, my problem does persist with Nx 19.6.2

pascalmann commented 2 weeks ago

I resolve my problem:

I had a .gitignore in the parent folder of my nx repo that affected the native glob method that is used by nx to retriev all project.json files in the workspace.
This is a really strange behavior that doesn't happen until 19.4.4.

mechanical-fish commented 2 weeks ago

This is the strangest bug ever. Inspired by @pascalmann I have tinkered with the .gitignore file, experimenting with deleting all or portions of it, and here's what I've been seeing:

The root directory of my project contains both a long .gitignore file and a long .eslintrc.js file. (You will see why I mention the latter in a second.) The command I'm trying to run is:

rm -rf .nx && NX_DAEMON=false NX_CACHE_PROJECT_GRAPH=false npx nx show projects --verbose

(I put the rm -rf .nx in there because i was getting irreproducible results and wanted to see if removing .nx's ability to save state would help with reproducibility.)

If I run this command in a fresh checkout of the codebase with Nx version 19.5.7 it exits successfully but prints nothing. (With Nx version 19.4.4 it prints a long list of projects, which is the expected behavior that I want.)

If I delete .gitignore or .eslintrc.js, or both, the command hangs. I cannot use Ctrl-C to break out of it, I have to run killall -9 node in another terminal to stop it.

If I make completely empty files called .gitignore and .eslintrc.js, the command still hangs.

At this point if I add a single line to .gitignore:

!.eslintrc.js

... a line which should do nothing because it is merely telling .gitignore to not ignore the file named .eslintrc.js... the nx show projects command exits successfully but prints nothing.

... I have no idea why the particular name .eslintrc.js for an empty file elicits such behavior. My project normally does install "@nx/eslint": "19.5.7" and "@nx/eslint-plugin": "19.5.7" as devDependencies but if I remove these I believe I still get the same behavior.

SerkanSipahi commented 2 weeks ago

Today we started our update cycle from v19.4.2 to nx v19.6.4 and I can confirm the same issues mentioned above:

Operating System: macOS sonoma Node: v18.20.4

Issues:

$ npx nx serve my-project

error: cannot find project 'my-project'

$ npx nx show projects
// empty list

SerkanSipahi commented 2 weeks ago

Update: I can confirm @pascalmann

I resolve my problem: I had a .gitignore in the parent folder of my nx repo that affected the native glob method that is used by nx to retriev all project.json files in the workspace. This is a really strange behavior that doesn't happen until 19.4.4.

My monorepo is located in ~/Projects/frontend/my-project and the parent .gitignore is located in my homedirectory (~/) because I am using the homedirectory for my dotfiles and ignore everything within .gitignore which is not related to the dotfiles.

I can also confirm this behavior does not happen until 19.4.4.

After removing the .gitignore from the parent directory (~/) everything works as accepted, so I was able to run npx nx show projects and npx nx serve my-project.

Additional notes: even the migrations fail when .gitignore was present in the parent directory. After removing it migrations was successfully.

lppedd commented 2 weeks ago

Remember you can also un-ignore paths using .nxignore. Patterns are merged with .gitignore ones.

SerkanSipahi commented 2 weeks ago

@lppedd The thing is that this behavior does not happen until v19.4.2 and as soon as you move to 19.6.2 and have a .gitignore in any parent directory this problem appears.

lppedd commented 2 weeks ago

Yeah definitely a bug, I was suggesting nxignore to avoid modifying gitignore.

pascalmann commented 1 week ago

.nxignore is not a solution. Nx should't consider files outside of his workspace.

mechanical-fish commented 1 week ago

@SerkanSipahi has identified my problem and why it is unique to my machine! This is what I get for keeping my dotfiles in Git.

Removing $HOME/.gitignore, which is two directories up the tree from my project directory, makes npx nx show projects work again. More specifically, my $HOME/.gitignore is set to ignore * (with a handful of exceptions), and removing the * from the file makes npx nx show projects work again.

SiKreuz commented 1 week ago

I also have a .gitignore with * in my home directory. I didn't get to try it out but I assume deleting it will also solve the issue for me. However, I'd like to keep that gitignore file there. I think the source for this issue is clear now. Do you need any additional information? @jaysoo

github-actions[bot] commented 1 day ago

This issue has been automatically marked as stale because more information has not been provided within 7 days. It will be closed in 21 days if no information is provided. If information has been provided, please reply to keep it active. Thanks for being a part of the Nx community! 🙏

nrwl / nx