search

npm / cli

the package manager for JavaScript
https://docs.npmjs.com/cli/
Other
8.48k stars 3.17k forks source link

[DOCS] npm doesn't install the latest tagged version if engines are invalid #7704

Open richardlau opened 3 months ago

richardlau commented 3 months ago

Is there an existing issue for this?

This issue exists in the latest npm version

Current Behavior

Running npm install @pkgjs/support or npx @pkgjs/support validate will install an outdated version of the package (0.0.2 instead of 0.0.6 which is also tagged latest).

Expected Behavior

The latest version of @pkgjs/support is installed/used.

Steps To Reproduce

npx

  1. Run npx @pkgjs/support validate with Node.js 22.5.1 (npm 10.8.2) Linux x64 
    $ npm version
{
npm: '10.8.2',
node: '22.5.1',
acorn: '8.11.3',
ada: '2.9.0',
ares: '1.32.1',
brotli: '1.1.0',
cjs_module_lexer: '1.2.2',
cldr: '45.0',
icu: '75.1',
llhttp: '9.2.1',
modules: '127',
napi: '9',
nbytes: '0.1.0',
nghttp2: '1.62.1',
nghttp3: '0.7.0',
ngtcp2: '1.3.0',
openssl: '3.0.13+quic',
simdjson: '3.9.4',
simdutf: '5.3.0',
sqlite: '3.46.0',
tz: '2024a',
undici: '6.19.2',
unicode: '15.1',
uv: '1.48.0',
uvwasi: '0.0.21',
v8: '12.4.254.21-node.16',
zlib: '1.3.0.1-motley-209717d'
}
$ npx @pkgjs/support validate
npm error could not determine executable to run
npm error A complete log of this run can be found in: /home/rlau/.npm/_logs/2024-08-06T15_47_54_908Z-debug-0.log
$

According to the debug log, @pkgjs/support@0.0.2 was used (where the error makes sense as there is no bin for version 0.0.2: https://registry.npmjs.org/@pkgjs%2fsupport).

$ cat /home/rlau/.npm/_logs/2024-08-06T15_47_54_908Z-debug-0.log
0 verbose cli /home/rlau/.nvm/versions/node/v22.5.1/bin/node /home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/bin/npm-cli.js
1 info using npm@10.8.2
2 info using node@v22.5.1
3 silly config load:file:/home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/npmrc
4 silly config load:file:/tmp/new/.npmrc
5 silly config load:file:/home/rlau/.npmrc
6 silly config load:file:/home/rlau/.nvm/versions/node/v22.5.1/etc/npmrc
7 verbose title npm exec @pkgjs/support validate
8 verbose argv "exec" "--" "@pkgjs/support" "validate"
9 verbose logfile logs-max:10 dir:/home/rlau/.npm/_logs/2024-08-06T15_47_54_908Z-
10 verbose logfile /home/rlau/.npm/_logs/2024-08-06T15_47_54_908Z-debug-0.log
11 silly logfile start cleaning logs, removing 1 files
12 silly logfile done cleaning log files
13 silly packumentCache heap:4345298944 maxSize:1086324736 maxEntrySize:543162368
14 http fetch GET 200 https://registry.npmjs.org/@pkgjs%2fsupport 592ms (cache miss)
15 verbose stack Error: could not determine executable to run
15 verbose stack     at getBinFromManifest (/home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/node_modules/libnpmexec/lib/get-bin-from-manifest.js:17:23)
15 verbose stack     at exec (/home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/node_modules/libnpmexec/lib/index.js:202:15)
15 verbose stack     at async Npm.exec (/home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/lib/npm.js:207:9)
15 verbose stack     at async module.exports (/home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/lib/cli/entry.js:74:5)
16 verbose pkgid @pkgjs/support@0.0.2
17 error could not determine executable to run
18 verbose cwd /tmp/new
19 verbose os Linux 5.14.0-284.11.1.el9_2.x86_64
20 verbose node v22.5.1
21 verbose npm  v10.8.2
22 verbose exit 1
23 verbose code 1
24 error A complete log of this run can be found in: /home/rlau/.npm/_logs/2024-08-06T15_47_54_908Z-debug-0.log
$

Problem does not occur with Node.js 22.4.1 and npm 10.8.1.

We first notice this happen in https://github.com/nodeshift/openshift-rest-client/actions/runs/10268248192/job/28410980498?pr=420 when attempting to add Node.js 22 to our existing GitHub Workflow.

npm install

  1. In an empty directory/new package...
  2. Run npm install @pkgjs/support. 
    
$ npm install @pkgjs/support

added 7 packages in 1s

1 package is looking for funding run npm fund for details $

3. Observe that `pkgjs/support@0.0.2` is installed:

$ npm ls new@ /tmp/new └── @pkgjs/support@0.0.2 $


Problem does not occur with Node.js 22.4.1 and npm 10.8.1.

### Environment

- npm: 10.8.2
- Node.js: 22.5.1
- OS Name: Linux
- System Model Name:
- npm config:
```ini
; "user" config from /home/rlau/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /home/rlau/.nvm/versions/node/v22.5.1/bin/node
; node version = v22.5.1
; npm local prefix = /tmp/new
; npm version = 10.8.2
; cwd = /tmp/new
; HOME = /home/rlau
; Run `npm config ls -l` to show all defaults.
richardlau commented 3 months ago

Keeping Node.js the same (22.5.1):

added 7 packages in 295ms

1 package is looking for funding run npm fund for details $


* npm 10.8.1 (expected behaviour):
```console
$ node /home/rlau/.nvm/versions/node/v22.4.1/lib/node_modules/npm/bin/npm-cli.js version
{
  npm: '10.8.1',
  node: '22.5.1',
  acorn: '8.11.3',
  ada: '2.9.0',
  ares: '1.32.1',
  brotli: '1.1.0',
  cjs_module_lexer: '1.2.2',
  cldr: '45.0',
  icu: '75.1',
  llhttp: '9.2.1',
  modules: '127',
  napi: '9',
  nbytes: '0.1.0',
  nghttp2: '1.62.1',
  nghttp3: '0.7.0',
  ngtcp2: '1.3.0',
  openssl: '3.0.13+quic',
  simdjson: '3.9.4',
  simdutf: '5.3.0',
  sqlite: '3.46.0',
  tz: '2024a',
  undici: '6.19.2',
  unicode: '15.1',
  uv: '1.48.0',
  uvwasi: '0.0.21',
  v8: '12.4.254.21-node.16',
  zlib: '1.3.0.1-motley-209717d'
}
$ node /home/rlau/.nvm/versions/node/v22.4.1/lib/node_modules/npm/bin/npm-cli.js install --dry-run @pkgjs/support
npm warn EBADENGINE Unsupported engine {
npm warn EBADENGINE   package: '@pkgjs/support@0.0.6',
npm warn EBADENGINE   required: { node: '^17 || ^16 || ^14 || ^13 || ^12 || ^11 || ^10.19' },
npm warn EBADENGINE   current: { node: 'v22.5.1', npm: '10.8.1' }
npm warn EBADENGINE }
add number-is-nan 1.0.1
add code-point-at 1.1.0
add camelcase 5.3.1
add p-try 2.2.0
add p-limit 2.3.0
add p-locate 4.1.0
add path-exists 4.0.0
add locate-path 5.0.0
add wrap-ansi 6.2.0
add color-name 1.1.4
add color-convert 2.0.1
add ansi-styles 4.3.0
add yargs-parser 18.1.3
add y18n 4.0.3
add which-module 2.0.1
add require-main-filename 2.0.0
add require-directory 2.1.1
add get-caller-file 2.0.5
add find-up 4.1.0
add decamelize 1.2.0
add cliui 6.0.0
add typedarray-to-buffer 3.1.5
add isexe 2.0.0
add builtins 1.0.3
add imurmurhash 0.1.4
add unique-slug 2.0.2
add punycode 2.3.1
add psl 1.9.0
add os-tmpdir 1.0.2
add has-flag 3.0.0
add ansi-regex 5.0.1
add is-fullwidth-code-point 3.0.0
add emoji-regex 8.0.0
add tweetnacl 0.14.5
add getpass 0.1.7
add ecc-jsbn 0.1.2
add jsbn 0.1.1
add dashdash 1.14.1
add bcrypt-pbkdf 1.0.2
add asn1 0.2.6
add sprintf-js 1.1.3
add jsbn 1.1.0
add smart-buffer 4.2.0
add ip-address 9.0.5
add socks 2.8.3
add tslib 1.14.1
add mimic-fn 2.1.0
add onetime 5.1.2
add mime-db 1.52.0
add verror 1.10.0
add core-util-is 1.0.2
add json-schema 0.4.0
add extsprintf 1.3.0
add sshpk 1.18.0
add jsbn 0.1.1
add jsprim 1.4.2
add assert-plus 1.0.0
add har-schema 2.0.0
add asynckit 0.4.0
add delayed-stream 1.0.0
add uuid 3.4.0
add tunnel-agent 0.6.0
add tough-cookie 2.5.0
add qs 6.5.3
add performance-now 2.1.0
add oauth-sign 0.9.0
add mime-types 2.1.35
add json-stringify-safe 5.0.1
add isstream 0.1.2
add is-typedarray 1.0.0
add http-signature 1.2.0
add har-validator 5.1.5
add form-data 2.3.3
add forever-agent 0.6.1
add extend 3.0.2
add combined-stream 1.0.8
add caseless 0.12.0
add aws4 1.13.1
add aws-sign2 0.7.0
add dezalgo 1.0.4
add debuglog 1.0.1
add util-deprecate 1.0.2
add string_decoder 1.1.1
add safe-buffer 5.1.2
add process-nextick-args 2.0.1
add isarray 1.0.0
add core-util-is 1.0.3
add end-of-stream 1.4.4
add asap 2.0.6
add promise 8.3.0
add promise-deferred 2.0.4
add is-callable 1.2.7
add retry 0.12.0
add err-code 2.0.3
add just-diff-apply 3.1.2
add just-diff 3.1.1
add json-parse-even-better-errors 2.3.1
add jsonparse 1.3.1
add minipass-sized 1.0.3
add encoding 0.1.13
add iconv-lite 0.6.3
add ms 2.1.2
add debug 4.3.6
add agent-base 6.0.2
add @tootallnate/once 1.1.2
add humanize-ms 1.2.1
add socks-proxy-agent 6.2.1
add negotiator 0.6.3
add is-lambda 1.0.1
add https-proxy-agent 5.0.1
add http-proxy-agent 4.0.1
add agentkeepalive 4.5.0
add minizlib 2.1.2
add minipass-json-stream 1.0.2
add minipass-fetch 1.4.1
add make-fetch-happen 9.1.0
add ignore-walk 3.0.4
add promise-retry 2.0.1
add npm-registry-fetch 11.0.0
add npm-packlist 2.2.2
add @npmcli/git 2.1.0
add indent-string 4.0.0
add clean-stack 2.2.0
add aggregate-error 3.1.0
add has-proto 1.0.3
add gopd 1.0.1
add has-property-descriptors 1.0.2
add define-data-property 1.1.4
add set-function-length 1.2.2
add get-intrinsic 1.2.4
add es-errors 1.3.0
add es-define-property 1.0.0
add object-keys 1.1.1
add has-symbols 1.0.3
add define-properties 1.2.1
add call-bind 1.0.7
add wide-align 1.1.5
add signal-exit 3.0.7
add object-assign 4.1.1
add has-unicode 2.0.1
add aproba 1.2.0
add readable-stream 2.3.8
add delegates 1.0.0
add set-blocking 2.0.0
add gauge 2.7.4
add ansi-regex 2.1.1
add is-fullwidth-code-point 1.0.0
add strip-ansi 3.0.1
add string-width 1.0.2
add console-control-strings 1.1.0
add are-we-there-yet 1.1.7
add validate-npm-package-name 3.0.0
add hosted-git-info 4.1.0
add abbrev 1.1.1
add function-bind 1.1.2
add hasown 2.0.2
add which 2.0.2
add request 2.88.2
add npmlog 4.1.2
add nopt 5.0.0
add env-paths 2.2.1
add concat-map 0.0.1
add balanced-match 1.0.2
add brace-expansion 1.1.11
add yallist 4.0.0
add json-buffer 3.0.1
add grapheme-splitter 1.0.4
add code-error-fragment 0.0.230
add promiseback 2.0.3
add object.assign 4.1.5
add node.extend 2.0.3
add is 3.3.0
add argparse 2.0.1
add safer-buffer 2.1.2
add tmp 0.0.33
add iconv-lite 0.4.24
add chardet 0.7.0
add restore-cursor 3.1.0
add type-fest 0.21.3
add through 2.3.8
add strip-ansi 6.0.1
add string-width 4.2.3
add rxjs 6.6.7
add run-async 2.4.1
add mute-stream 0.0.8
add lodash 4.17.21
add figures 3.2.0
add external-editor 3.1.0
add cli-width 3.0.0
add cli-cursor 3.1.0
add ansi-escapes 4.3.2
add wrappy 1.0.2
add resolve-alpn 1.2.1
add quick-lru 5.1.1
add pump 3.0.0
add mimic-response 1.0.1
add keyv 4.5.4
add http-cache-semantics 4.1.1
add get-stream 5.2.0
add clone-response 1.0.3
add undici-types 6.13.0
add @types/node 22.1.0
add @types/keyv 3.1.4
add @types/http-cache-semantics 4.0.4
add defer-to-connect 2.0.1
add responselike 2.0.1
add p-cancelable 2.1.1
add lowercase-keys 2.0.0
add http2-wrapper 1.0.3
add decompress-response 6.0.0
add mimic-response 3.1.0
add cacheable-request 7.0.4
add normalize-url 6.1.0
add cacheable-lookup 5.0.4
add @types/responselike 1.0.3
add @types/cacheable-request 6.0.3
add @szmarczak/http-timer 4.0.6
add @sindresorhus/is 4.6.0
add path-is-absolute 1.0.1
add once 1.4.0
add inherits 2.0.4
add inflight 1.0.6
add fs.realpath 1.0.0
add universalify 0.1.2
add jsonfile 4.0.0
add graceful-fs 4.2.11
add color-name 1.1.3
add color-convert 1.9.3
add supports-color 5.5.0
add escape-string-regexp 1.0.5
add ansi-styles 3.2.1
add @gar/promisify 1.1.3
add unique-filename 1.1.1
add tar 6.2.1
add minipass 5.0.0
add ssri 8.0.1
add promise-inflight 1.0.1
add p-map 4.0.0
add mkdirp 1.0.4
add minipass-pipeline 1.2.4
add minipass-flush 1.0.5
add minipass-collect 1.0.2
add minipass 3.3.6
add lru-cache 6.0.0
add fs-minipass 2.1.0
add chownr 2.0.0
add @npmcli/move-file 1.1.2
add @npmcli/fs 1.1.1
add write-file-atomic 3.0.3
add rimraf 3.0.2
add read-cmd-shim 2.0.0
add cmd-shim 4.1.0
add regenerator-runtime 0.14.1
add js-tokens 4.0.0
add @babel/helper-validator-identifier 7.24.7
add picocolors 1.0.1
add @babel/highlight 7.24.7
add leven 3.1.0
add jsonpointer 4.1.0
add json-to-ast 2.1.0
add core-js 3.38.0
add chalk 2.4.2
add @babel/runtime 7.25.0
add @babel/code-frame 7.24.7
add uri-js 4.4.1
add json-schema-traverse 0.4.1
add fast-json-stable-stringify 2.1.0
add fast-deep-equal 3.1.3
add infer-owner 1.0.4
add node-gyp 7.1.2
add @npmcli/promise-spawn 1.3.2
add @npmcli/node-gyp 1.0.3
add minimatch 3.1.2
add glob 7.2.3
add npm-normalize-package-bin 1.0.1
add npm-bundled 1.1.2
add walk-up-path 1.0.0
add treeverse 1.0.4
add semver 7.6.3
add readdir-scoped-modules 1.1.0
add read-package-json-fast 1.2.2
add promise-call-limit 1.0.2
add promise-all-reject-late 1.0.1
add parse-conflict-json 1.1.1
add pacote 11.3.5
add read-package-json-fast 2.0.3
add npm-pick-manifest 6.1.1
add npm-package-arg 8.1.5
add npm-install-checks 4.0.0
add mkdirp-infer-owner 2.0.0
add json-stringify-nice 1.1.4
add common-ancestor-path 1.0.1
add cacache 15.3.0
add bin-links 2.3.0
add @npmcli/run-script 1.8.6
add read-package-json-fast 2.0.3
add @npmcli/name-from-folder 1.0.1
add @npmcli/map-workspaces 0.0.0-pre.1
add @npmcli/installed-package-contents 1.0.7
add yargs 15.4.1
add normalize-url 5.3.1
add loggerr 3.3.0
add has-flag 4.0.0
add color-name 1.1.4
add color-convert 2.0.1
add supports-color 7.2.0
add ansi-styles 4.3.0
add chalk 4.1.2
add json-file-plus 3.3.1
add js-yaml 4.1.0
add inquirer 7.3.3
add has-flag 4.0.0
add color-name 1.1.4
add color-convert 2.0.1
add supports-color 7.2.0
add ansi-styles 4.3.0
add chalk 4.1.2
add got 11.8.6
add fs-extra 8.1.0
add better-ajv-errors 0.6.7
add ajv 6.12.6
add @npmcli/arborist 0.0.0
add @pkgjs/support 0.0.6

added 323 packages in 4s

38 packages are looking for funding
  run `npm fund` for details
$
richardlau commented 3 months ago

Also I don't believe it's cache related either:

$ rm -rf /tmp/empty-cache
$ node /home/rlau/.nvm/versions/node/v22.5.1/lib/node_modules/npm/bin/npm-cli.js install --dry-run --cache /tmp/empty-cache @pkgjs/support
add punycode 2.3.1
add uri-js 4.4.1
add json-schema-traverse 0.4.1
add fast-json-stable-stringify 2.1.0
add fast-deep-equal 3.1.3
add ajv 6.12.6
add @pkgjs/support 0.0.2

added 7 packages in 598ms

1 package is looking for funding
  run `npm fund` for details
$
richardlau commented 3 months ago

And if anyone is wondering, problem still occurs on freshly released Node.js 22.6.0 (still with npm 10.8.2).

milaninfy commented 3 months ago

@richardlau npm prioritises package that match engines over the ones which don't, if no specific version provided. https://github.com/npm/npm-pick-manifest/pull/33

~/workarea/rep/testing $ npx @pkgjs/support@latest validate
Need to install the following packages:
@pkgjs/support@0.0.6
Ok to proceed? (y) y
<...some output...>

Running npm install @pkgjs/support@latest

~/workarea/rep/testing $ npm -v      
10.8.2
~/workarea/rep/testing $ npm install @pkgjs/support@latest
npm warn EBADENGINE Unsupported engine {
npm warn EBADENGINE   package: '@pkgjs/support@0.0.6',
npm warn EBADENGINE   required: { node: '^17 || ^16 || ^14 || ^13 || ^12 || ^11 || ^10.19' },
npm warn EBADENGINE   current: { node: 'v22.5.1', npm: '10.8.2' }
npm warn EBADENGINE }
npm warn deprecated debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs
npm warn deprecated readdir-scoped-modules@1.1.0: This functionality has been moved to @npmcli/fs
npm warn deprecated har-validator@5.1.5: this library is no longer supported
npm warn deprecated npmlog@4.1.2: This package is no longer supported.
npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
npm warn deprecated are-we-there-yet@1.1.7: This package is no longer supported.
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
npm warn deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm warn deprecated gauge@2.7.4: This package is no longer supported.
npm warn deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

added 323 packages, and audited 324 packages in 3s

38 packages are looking for funding
  run `npm fund` for details

9 vulnerabilities (7 moderate, 2 high)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
~/workarea/rep/testing $ npm ls
testing@1.0.0 /Users/milaninfy/workarea/rep/testing
└── @pkgjs/support@0.0.6

richardlau commented 3 months ago

@richardlau npm prioritises package that match engines over the ones which don't, if no specific version provided. npm/npm-pick-manifest#33

That's an unexpected, breaking and undocumented change in behaviour in a semver-patch release (10.8.2 vs 10.8.1).

npm install @pkgjs/support

should be "f) a <name> that has a "latest" tag satisfying (e)".

https://github.com/npm/cli/blob/4e81a6a4106e4e125b0eefda042b75cfae0a5f23/docs/lib/content/commands/npm-install.md?plain=1#L25-L36

richardlau commented 3 months ago

We've also hit this in the Node.js Jenkins CITGM CI. 1) citgm@9.2.2 's package.json did not include Node.js 22, which meant that npm install citgm was installing citgm@8.1.0 (which didn't have the engines added) when used with Node.js 22 release proposals. 2) Even with the updated engines field, the CI won't match for Node.js built from main (23.0.0-pre).

I have updated Node.js' Jenkins jobs to default the "which CITGM" parameter to citgm@latest instead of plain citgm but I really am of the opinion that this is a breaking change and shouldn't have been made in a semver-patch npm release.