`nx release` updating lock file un-syncs lock file with package.json

m-thalmann commented 7 months ago

Current Behavior

When executing the nx release command the package-lock.json file is updated (which is expected). But it also removes peer-dependencies which are actually in use. Therefore subsequent npm ci calls fail, because the package.json and the package-lock.json file are not in sync anymore (executing npm i adds the dependencies back to the lock file).

In my case this refers to the following transitive (peer-)dependencies of @nx/js:

@types/node
typescript
undici-types

Expected Behavior

The nx release command updates the package-lock file, but only updates the versions of the newly release packages and does not remove any used (peer-)dependencies

GitHub Repo

https://github.com/m-thalmann/codestyle-packages/

Steps to Reproduce

Create a Nx Package-Based project (pure JS, no Typescript)
Create a package and commit the changes

Setup Nx Release

I used the following config:

{
"release": {
  "projects": ["*"],
  "projectsRelationship": "independent",
  "version": {
    "conventionalCommits": true
  },
  "git": {
    "commitMessage": "build(release): create new versions of packages"
  },
  "changelog": {
    "projectChangelogs": {
      "renderOptions": {
        "authors": false
       }
    }
  }
}
}

Run nx release --skip-publish --first-release (or omit the skip-publish, does not matter) (its also not dependent on the first release flag)
Inspect the create commit: The package-lock should be updated with the above mentioned dependencies removed
Run npm ci -> this will fail

Nx Report

Node   : 20.11.0
OS     : win32-x64
npm    : 10.2.4

nx (global)    : 18.0.3
nx             : 18.0.5
@nx/js         : 18.0.5
@nx/workspace  : 18.0.5
@nx/devkit     : 18.0.5
@nrwl/tao      : 18.0.5
typescript     : 5.3.3

Failure Logs

No response

Package Manager Version

No response

Operating System

[ ] macOS
[X] Linux
[X] Windows
[ ] Other (Please specify)

Additional Information

This issue occurs on both our local windows developer machines and the linux build servers.

I tried to find the root cause but I was not able to reproduce this issue when running the npm install --package-lock-only command (which is used by the update lock file function in our case). I've also tried executing it from a custom JS script but this did also not reproduce the above mentioned issue.

Maybe it is something in the packages/js/src/generators/release-version/utils/update-lock-file.ts file but I have no idea how to start.

lancempoe commented 7 months ago

I am getting a similar error. I'll note that it also fails when setting skipLockFileUpdate as noted in the error logs shown below. We are blocked now :( Maybe there is a version of 18 that I can use that doesn't have this error?

I'll also note that there is no documentation for skipLockFileUpdate in the readme so adding something there might be nice.

NX Updating npm lock file npm ERR! Cannot read properties of null (reading 'package') npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-03-02T06_23_02_573Z-debug-0.log NX Error updating lock file with command 'npm install --package-lock-only' Verify that 'npm install --package-lock-only' succeeds when run from the workspace root. To configure a string of arguments to be passed to this command, set the 'release.version.generatorOptions.installArgs' property in nx.json. To ignore install lifecycle scripts, set 'release.version.generatorOptions.installIgnoreScripts' to true in nx.json. To disable this step entirely, set 'release.version.skipLockFileUpdate' to true in nx.json.

Here are my settings:

"release": { "projectsRelationship": "independent", "version": { "conventionalCommits": true }, "changelog": { "projectChangelogs": true } } },

Relavent package versions:

"nx": "^18.0.6"
"@nrwl/cli": "^15.9.7",
"@nx/eslint-plugin": "^18.0.6",
"@nx/devkit": "^18.0.6",
"@nx/jest": "^18.0.6",
"@nx/js": "^18.0.6",
"@nx/node": "^18.0.6",

The final note is that it changes our internal dependencies from * to a direct number and that breaks the lock file

ryan-mcginty-alation commented 7 months ago

I am also experiencing a similar issue. npm ci fails after publish run due to peer dependency issues after release command updates the lock file incorrectly.

Checking to see if https://github.com/s1seven/schema-tools/issues/253 fixes it

ryan-mcginty-alation commented 7 months ago

Still not working for me, below is the error on CI/CD. It requires re-running npm i to regenerate the lock file to fix it. Using Nx 18.0.8

npm WARN ERESOLVE overriding peer dependency
npm WARN While resolving: @vitejs/plugin-react@3.1.0
npm WARN Found: vite@5.0.10
npm WARN node_modules/vite
npm WARN   dev vite@"5.0.10" from the root project
npm WARN   15 more (@joshwooding/vite-plugin-react-docgen-typescript, ...)
npm WARN 
npm WARN Could not resolve dependency:
npm WARN peer vite@"^4.1.0-beta.0" from @vitejs/plugin-react@3.1.0
npm WARN node_modules/@storybook/react-vite/node_modules/@vitejs/plugin-react
npm WARN   @vitejs/plugin-react@"^3.0.1" from @storybook/react-vite@7.6.17
npm WARN   node_modules/@storybook/react-vite
npm WARN 
npm WARN Conflicting peer dependency: vite@4.5.2
npm WARN node_modules/vite
npm WARN   peer vite@"^4.1.0-beta.0" from @vitejs/plugin-react@3.1.0
npm WARN   node_modules/@storybook/react-vite/node_modules/@vitejs/plugin-react
npm WARN     @vitejs/plugin-react@"^3.0.1" from @storybook/react-vite@7.6.17
npm WARN     node_modules/@storybook/react-vite
npm ERR! code EUSAGE
npm ERR! 
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! Missing: next@14.1.3 from lock file
npm ERR! Missing: @next/env@14.1.3 from lock file
npm ERR! Missing: @next/swc-darwin-arm64@14.1.3 from lock file
npm ERR! Missing: @next/swc-darwin-x64@14.1.3 from lock file
npm ERR! Missing: @next/swc-linux-arm64-gnu@14.1.3 from lock file
npm ERR! Missing: @next/swc-linux-arm64-musl@14.1.3 from lock file
npm ERR! Missing: @next/swc-linux-x64-gnu@14.1.3 from lock file
npm ERR! Missing: @next/swc-linux-x64-musl@14.1.3 from lock file
npm ERR! Missing: @next/swc-win[32](arm64-msvc@14.1.3 from lock file
npm ERR! Missing: @next/swc-win32-ia32-msvc@14.1.3 from lock file
npm ERR! Missing: @next/swc-win32-x64-msvc@14.1.3 from lock file
npm ERR! Missing: @swc/helpers@0.5.2 from lock file
npm ERR! Missing: busboy@1.6.0 from lock file
npm ERR! Missing: styled-jsx@5.1.1 from lock file
npm ERR! Missing: streamsearch@1.1.0 from lock file
npm ERR! Missing: @storybook/addons@7.6.17 from lock file
npm ERR! Missing: @storybook/api@7.6.17 from lock file
npm ERR! Missing: client-only@0.0.1 from lock file
npm ERR! 
npm ERR! Clean install a project
npm ERR! 
npm ERR! Usage:
npm ERR! npm ci
npm ERR! 
npm ERR! Options:
npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm ERR! 
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR! 
npm ERR! Run "npm help ci" for more info

ryan-mcginty-alation commented 7 months ago

Finally figured out a workaround in programmatic api:

const {workspaceVersion, projectsVersionData} = await releaseVersion({
    specifier: options.version,
    dryRun: options.dryRun,
    verbose: options.verbose,
    gitCommit: false, // Commit will be created in the releaseChangelog step
    gitTag: false, // Tag will be created in the releaseChangelog step
  });

  // workaround for https://github.com/nrwl/nx/issues/22066
  console.log('un-stage and checkout package-lock.json');
  execSync('git restore --staged package-lock.json', {stdio: 'inherit'});
  execSync(`git checkout package-lock.json`, {stdio: 'inherit'});

  await releaseChangelog({
    gitCommitMessage: `chore: ${options.issueKey}: release version {version}`,
    versionData: projectsVersionData,
    version: workspaceVersion,
    dryRun: options.dryRun,
    verbose: options.verbose,
  });

ThePlenkov commented 6 months ago

Please provide the option to disable updating package-lock.json with release. It's basically breaking things. After this command next npm install removes dozens of depencies from my project and it stops working locally. The only way to fix - is to remove node_modules and package-lock.json and reinstall. This is very inconvenient..

ThePlenkov commented 6 months ago

OK just found it:

  "release": {
    "version": {
      "generatorOptions": {
        "skipLockFileUpdate": true
      }
    }
  }

made a trick in nx.json. Now it's good

bentinata commented 5 months ago

@ThePlenkov Wouldn't this make lockfile unsync with package.json since nx release is updating version in package.json, but not in the package-lock.json?

ThePlenkov commented 5 months ago

Absolutely correct, that's what I fix with usage of workspaces + npm install package lock only in post install script. Works like a charm

elliewintram commented 4 months ago

I was also experiencing this issue whereby the nx release command was removing peer deps from the package-lock.json, resulting in npm ci failing on subsequent builds due to the package.json and package-lock.json being out of sync (using node v20, npm v10).

For me it seemed to be caused by the fact that nx release calls npm install with the legacy-peer-deps set to true, seen in the source code here.

Following the docs for npm ci, I added legacy-peer-deps=true to the .npmrc file to ensure peer deps are not installed on any npm install command, which is the behaviour of older npm versions.

NOTE: If you create your package-lock.json file by running npm install with flags that can affect the shape of your dependency tree, such as --legacy-peer-deps or --install-links, you must provide the same flags to npm ci or you are likely to encounter errors. An easy way to do this is to run, for example, npm config set legacy-peer-deps=true --location=project and commit the .npmrc file to your repo.

This removed the peer deps in my package-lock.json and resulted in no further changes to the package-lock.json on nx release.

Question for the nx team - is process.env.npm_config_legacy_peer_deps ??= 'true'; going to be removed in a future version based on the comment in the source code?

arichard-info commented 3 months ago

Hello everyone,

Like you, I have the package-lock.json problem, I'm also looking for solutions. But something struck me in the discussions: Is it really an npm install that is performed by nx release to re-synchronise the package-lock.json?

https://github.com/nrwl/nx/blob/master/packages/nx/src/utils/package-manager.ts#L160

Is this deliberate? It's likely to update dependencies that aren't affected by the release, isn't it? I was right to look into this because I thought that the package-lock.json was updated in a much more meticulous way, with only the versions affected.

If that's the case, it's highly inadvisable to run this command in a CI, isn't it?

nrwl / nx