search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.61k stars 979 forks source link

Npm packages are unexpectedly downloaded from the internal registry #10227

Open drik98 opened 1 month ago

drik98 commented 1 month ago

Is there an existing issue for this?

Package ecosystem

npm

Package manager version

pnpm@8.15.1

Language version

node v20.12.2

Manifest location and content before the Dependabot update

/package.json /pnpm-lock.yaml

dependabot.yml content

version: 2
registries:
  npm-private:
    type: npm-registry
    url: <internal-nexus>/repository/npm-private
    token: ${{NEXUS_TOKEN}}
updates:
  - package-ecosystem: npm
    directory: /
    schedule:
      interval: weekly
    reviewers:
      - gitlabuser1
      - gitlabuser2
      - gitlabuser3
      - gitlabuser4
      - gitlabuser5
      - gitlabuser6
      - gitlabuser7
      - gitlabuser8

.npmrc:

@internal:registry=https://<internal-nexus>/repository/npm-private/

Updated dependency

An internal one from the self hosted nexus.

What you expected to see, versus what you actually saw

Given that I have not set replaces-base on the configured registry npm-private to true and also only set the scoped registry (@internal:registry=...) and not registry=... in the .npmrc file I would expect that while the updated internal dependency is pulled from the self hosted nexus all other (public) packages are pulled from the official npm registry (https://registry.npmjs.org). Instead I can observe with debug log enabled that all packages are pulled from the internal nexus (see logs for more details):flag_vi:

[35m[2024-07-09 05:08:09 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/@vueuse%2Fcore'
[35m[2024-07-09 05:08:10 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/@vueuse%2Fcore', status: 200

This results (I think) in a really slow execution of the version update (up to 10 minutes per updated dependency/creating the Merge request). This sometimes causes builds to timeout and also results in a lot of used build minutes.

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

Log output

[35m[2024-07-09 05:07:54 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Evaluating dependency type rule {:dependency_type=>"direct"} for @internal/package
[35m[2024-07-09 05:07:54 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Rule evaluated to 'true'
[32m[2024-07-09 05:07:54 +0000] INFO  -- [0m[dep-update: redacted/repository=>npm] Processing dependency [1m@internal/package[0m
[35m[2024-07-09 05:07:54 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/@internal%2Fpackage'
[35m[2024-07-09 05:07:55 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/@internal%2Fpackage', status: 200
[35m[2024-07-09 05:07:55 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/@internal%2Fpackage/2.0.1-990ae8fd-1364618120-dev'
[35m[2024-07-09 05:07:55 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/@internal%2Fpackage/2.0.1-990ae8fd-1364618120-dev', status: 400
...
[35m[2024-07-09 05:08:08 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Using following ignore version conditions: []
[32m[2024-07-09 05:08:08 +0000] INFO  -- [0m[dep-update: redacted/repository=>npm]   updating [1m@internal/package: 2.0.1-dadc5a59-1308155479-dev[0m => [1m2.0.1-990ae8fd-1364618120-dev[0m
...
[35m[2024-07-09 05:08:09 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/@vueuse%2Fcore'
[35m[2024-07-09 05:08:10 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/@vueuse%2Fcore', status: 200
[35m[2024-07-09 05:08:10 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/axios'
[35m[2024-07-09 05:08:11 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/axios', status: 200
[35m[2024-07-09 05:08:11 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/dayjs'
[35m[2024-07-09 05:08:11 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/dayjs', status: 200
[35m[2024-07-09 05:08:11 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/oidc-client-ts'
[35m[2024-07-09 05:08:12 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://<internal-nexus>/repository/npm-private/oidc-client-ts', status: 200
[35m[2024-07-09 05:08:12 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://<internal-nexus>/repository/npm-private/pinia'
... (and all the other dependencies)

[35m[2024-07-09 05:16:26 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser1
[35m[2024-07-09 05:16:27 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser2
[35m[2024-07-09 05:16:27 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser3
[35m[2024-07-09 05:16:27 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser4
[35m[2024-07-09 05:16:28 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser5
[35m[2024-07-09 05:16:28 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser6
[35m[2024-07-09 05:16:28 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser7
[35m[2024-07-09 05:16:28 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] Running user search for gitlabuser8
[35m[2024-07-09 05:16:29 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://registry.npmjs.org/@internal%2Fpackage/latest'
[35m[2024-07-09 05:16:30 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://registry.npmjs.org/@internal%2Fpackage/latest', status: 404
[35m[2024-07-09 05:16:30 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Performing http :get request to 'https://registry.npmjs.org/@internal%2Fpackage'
[35m[2024-07-09 05:16:30 +0000] DEBUG -- [0m[dep-update: redacted/repository=>npm] [core] Received response from 'https://registry.npmjs.org/@internal%2Fpackage', status: 404
[32m[2024-07-09 05:16:32 +0000] INFO  -- [0m[dep-update: redacted/repository=>npm]   skipping merge request [1mhttps://gitlab.com/redacted/repository/-/merge_requests/954[0m update, mr has no conflicts and rebase strategy is set to [1mauto[0m

Smallest manifest that reproduces the issue

No response

drik98 commented 1 month ago

Note: I am using dependabot on gitlab through dependabot-gitlab in the standalone version which uses dependabot-core. I created an issue there but was pointed here as the logic regarding fetching the dependencies using the configured registries comes from the dependabot-core.

EzraBrooks commented 2 hours ago

I'm seeing this issue as well - on GitHub, not GitLab - and it causes my Dependabot jobs to time out 100% of the time because I have a lot of dependencies and it's erroneously searching for them on GitHub Packages when it should be searching for them on npmjs.org.

drik98 commented 1 hour ago

I'm seeing this issue as well - on GitHub, not GitLab - and it causes my Dependabot jobs to time out 100% of the time because I have a lot of dependencies and it's erroneously searching for them on GitHub Packages when it should be searching for them on npmjs.org.

„Great“ to hear that this is not a gitlab only issue. Are you able to create a github only repro?