Lerna seems to not found packages after upgrade

dboulardsc commented 4 months ago

Current Behavior

Updated from v7 to v8 with the following config

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "1.0.0",
  "npmClient": "yarn"
}

Executing list returns to have found 0 packages. Meanwhile :

Occurs only on this very computer. My other one is okay with the upgrade. Also ok for every other colleague in my team. Also ok for github ci/cd.
yarn workspace info returns the correct amount of packages
npx lerna clean seems to find the various packages to remove the node_modules
Tried on a new-from-scratch project and same behaviour.
Manually adding packages in lerna.json or root package.json doesnt change anything
yarn cache clean doesnt change anything
rm ~/.yarn doesnt change anything

Also, seems quite linked to https://github.com/lerna/lerna/issues/4018

Expected Behavior

List the packages my monorepo contains.

Steps to Reproduce

Started a project from scratch and can reproduce the issue.

yarn init
yarn add lerna
lerna init
mkdir packages
lerna create patata
lerna create patato

root package.json :

{
  "private": true,
  "workspaces": [
    "packages/*"
  ],
  "name": "lernatest",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "lerna": "^8.1.7"
  }
}

first package :

{
  "name": "patata",
  "version": "1.0.0",
  "main": "lib/patata.js",
  "scripts": {
    "build": "echo \"build patata\""
  },
  "dependencies": {
    "patato": "1.0.0"
  }
}

second package :

{
  "name": "patato",
  "version": "1.0.0",
  "main": "lib/patato.js",
  "scripts": {
    "build": "echo \"build patato\""
  }
}

Failure Logs / Configuration

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "1.0.0",
  "npmClient": "yarn"
}

Environment

lerna notice cli v8.1.7

 Environment info:

  System:
    OS: Linux 5.15 Manjaro Linux
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
  Binaries:
    Node: 20.16.0 - ~/.nvm/versions/node/v20.16.0/bin/node
    Yarn: 1.22.22 - /usr/bin/yarn
    npm: 10.8.1 - ~/.nvm/versions/node/v20.16.0/bin/npm
  Utilities:
    Git: 2.45.2 - /usr/bin/git
  npmPackages:
    lerna: ^8.1.7 => 8.1.7

JamesHenry commented 3 months ago

@dboulardsc Sorry to hear that, issues that only happen under some niche circumstances are the worst, because I naturally can't repro either:

Some ideas to try (in order from simplest to most effort, not necessarily most likely to be the reason):

Running npm ls nx to check you have 19.5.6 or similar of the nx dependency of lerna installed and not an old version
Running yarn nx reset and retrying the commands
Is the directory you are working in nested under another git directory? It's unusual and almost always accidental, but I believe if a parent directory had a .git directory it could cause weird issues (and that would be the kind of thing that could happen on one specific user's machine). The same is true for parent package.json and lock files I believe.
As well as just trying to find .git and package.json etc directories in the hierarchy, in general, please try just moving the lerna workspace to a different location on you machine and see if it changes anything.
Uninstalling and reinstalling yarn classic altogether

GregoireBellon commented 3 months ago

I face the same issue with npm and lerna 8.1.8.

I have two packages (@package/front and @package/back) that both share a dependency (@package/shared)

npx nx graph shows the dependency relationship, but the command lerna run build --scope @package/back--include-dependencies does not build my shared dependency.

The output is :

lerna notice cli v8.1.8
lerna notice filter including "@package/back"
lerna notice filter including dependencies
lerna info filter [ '@package/back' ]

> @package/back:build

> build
> nest build
... then fails irremediably

The package.json files are very similar to @dboulardsc 's ones, but the dependencies are declared like this :

  "dependencies": {
    "@package/shared": "*"
  }

The command used to work, and no package.json of the monorepo changed significantly, so I guess that the issue could have appeared in lerna 8.1.8 ?

ArthurG94 commented 3 months ago

I'm experiencing the same issue with Node 20.16.0 (via nvm), npm, and Lerna >= 8.1.3.

If I roll back to version 8.1.2, npx lerna list works again:

npm i -D lerna@latest
npx lerna list
#  lerna notice cli v8.1.8
#  lerna success found 0 packages

npm i -D lerna@8.1.3
npx lerna list
#  lerna notice cli v8.1.3
#  lerna success found 0 packages

npm i -D lerna@8.1.2
npx lerna list
#  lerna notice cli v8.1.2
#  @company/xxx-client
#  @company/xxx-server
#  lerna success found 2 packages

Project Architecture

client

package.json

{
  "name": "@company/xxx-client",
  "version": "x.x.x"
  "scripts": {
      "eslint": "eslint --fix \"src/**/*.ts\" ",
      ...
  }
}

server

package.json

{
  "name": "@company/xxx-server",
  "version": "x.x.x",
  "scripts": {
      "eslint": "eslint --fix ./src/**/* --ext .ts",
      ...
  }
}

package.json

{
  "name": "@company/xxx",
  "private": true,
  "workspaces": [
      "server",
      "client"
  ],
  "scripts": {
      "eslint": "npx lerna run eslint",
      ...
  }
}

lerna.json

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "5.4.1"
}

Environment

  System:
    OS: Linux 6.8 Ubuntu 24.04 LTS 24.04 LTS (Noble Numbat)
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 20.16.0 - ~/.nvm/versions/node/v20.16.0/bin/node
    npm: 10.8.1 - ~/.nvm/versions/node/v20.16.0/bin/npm
  Utilities:
    Git: 2.43.0 - /usr/bin/git
  npmPackages:
    lerna: ^8.1.8 => 8.1.8

loganzartman commented 1 month ago

same issue here--showing zero packages. I'm defining packages via yarn classic workspaces. I'm stuck on yarn 1.18.0 (sadly).

I agree with @ArthurG94: I decremented the lerna version and tested each one. It starts working again at 8.1.2. I would consider this a regression between 8.1.2 and 8.1.3. This is the only thing I changed, and I was able to repro this consistently when changing versions. I have only the minimal lerna init lerna config:

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "0.0.0",
  "npmClient": "yarn"
}

kaiz-io commented 3 weeks ago

I am running to this also and trying to nail this down. I have two machines that are doing this both built the same way using ansible, so I feel like I should be able to replicate this on more machines. I have tried node, npm, npx from apt and from nvm.

It works in a docker environment with the same sample monorepo and node_modules installed from the host machine. I am starting to grasp at straws, but this weekend will go about creating a 3rd machine to see if I can confirm what is causing the issue.

By chance @loganzartman @dboulardsc are you running btrfs on your filesystem?

kaiz-io commented 3 weeks ago

Edit: Looks like it is related to this So I found that I had I am using dotfiles that has a ~/.gitignore which has a * and then I add ! for files that I want my dotfiles to track. This does not affect my any other git repo commands tracking only lerna is affected by this.

~ 
|>.gitignore                                                      < this .gitignore has an affect on lerna
|_code
|_____|_playground
|__________________|_lerna                             < directory where I have my simple lerna project
__________________________|_lerna.json
__________________________|_.gitignore          < standard

~/.gitingore

*
!.gitignore
.gitconfig
!code/playground/lerna/* < have to add this line for lerna to work.
....

~/code/playground/lerna/.gitignore

node_modules/

JamesHenry commented 3 weeks ago

Thank you @kaiz-io I have a feeling that the .gitignore is going to be the explanation in most if not all cases.

If you .gitignore a package, then it will be excluded from the project graph construction. This is powered by nx core behind the scenes (to allow for free integrations with Nx Cloud and Nx Console, and benefit from all that work within Lerna). There is a .nxignore file that has higher precedence than .gitignore, so if you want to keep the package git ignored, you can add a .nxignore file with the negative pattern in instead:

Please let me know if that works for you.

Please could @loganzartman @GregoireBellon @ArthurG94 @dboulardsc confirm if they also have a git ignore involved and if the negative pattern in .nxignore (i.e. forcing it to be considered, despite being git ignored) works?

loganzartman commented 3 weeks ago

I'll get back to you when I have time to check the fix, but I'll say that I do have a .gitignore in my home directory. So:

/home/logan
    .git
    .gitignore
    project
        .git
        .gitignore

The top gitignore is used to exclude everything but dotfiles so that I can VC them.

However there is no reason the top .gitignore should apply to nx, because the lerna project itself is in a git repo. Git does not work the way that nx is here, otherwise this setup would not work and no one would use it. So maybe this is a bug in nx rather than lerna, but I don't think it's user error.

ArthurG94 commented 3 weeks ago

Hi, I can't check the fix either, but I confirm that I have a .gitignore file in my home folder that excludes everything except a few config files.

JamesHenry commented 3 weeks ago

@loganzartman Thanks a lot for that overview, I was able to reproduce and we are working on a fix

JamesHenry commented 3 weeks ago

I do think that .gitignore is the common theme here when folks are saying packages are not found, and there are two main things to summarize at this point:

If you are impacted in the same way as @loganzartman, where your lerna workspace is nested inside another git repo that ignores all its children, you will need to wait for the bugfix
If you are impacted because you are trying to version and publish a package that you have intentionally gitignored via the gitignore file in the root of the lerna workspace itself, you should add an entry to .ignore at the root of the lerna workspace that negates that ignore.

E.g. if you have:

.gitignore

# Git ignore foo
packages/foo

You will need:

.ignore

# Cancel out the fact that this is ignored by git when it comes to considering it for project graph construction
!packages/foo

JamesHenry commented 3 weeks ago

I have updated my previous post, apparently we support a file just called .ignore as well which may be preferable to you. .ignore or .nxignore should work for the purposes of negating the git ignore's influence on the project graph construction

loganzartman commented 2 weeks ago

Appreciate the info @JamesHenry, is there an issue or PR to track for the fix? I assume on the nx side?

JamesHenry commented 1 week ago

Yes, because it's not hard blocked (the ignore file usage works) it's going into our next cooldown cycle, rest assured I will post here when it's available

lerna / lerna