search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.62k stars 987 forks source link

Private npm/yarn repositories no longer work (private_source_timed_out) #10218

Open mbiggs-gresham opened 1 month ago

mbiggs-gresham commented 1 month ago

Is there an existing issue for this?

Package ecosystem

npm (yarn)

Package manager version

yarn v4.3.1

Language version

typescript

Manifest location and content before the Dependabot update

No response

dependabot.yml content

No response

Updated dependency

No response

What you expected to see, versus what you actually saw

Private npm repositories no longer work for regular updates. Every attempt now results in a private_source_timed_out since ~12th July. Based on the timestamps in the log snippet below I can't see how anything is timing out that quickly.

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

2024-07-16T05:27:39.4318521Z updater | 2024/07/16 05:27:39 INFO <job_856353230> Starting update group for 'typescript'
2024-07-16T05:27:39.4515571Z updater | 2024/07/16 05:27:39 INFO <job_856353230> Updating the /xxxxxxxx directory.
2024-07-16T05:27:40.2724807Z updater | 2024/07/16 05:27:40 INFO <job_856353230> Checking if typescript 5.5.3 needs updating
2024-07-16T05:27:40.3246186Z   proxy | 2024/07/16 05:27:40 [033] POST /update_jobs/856353230/record_update_job_error
2024-07-16T05:27:40.3248407Z   proxy | 2024/07/16 05:27:40 [033] 204 /update_jobs/856353230/record_update_job_error
2024-07-16T05:27:40.3257218Z updater | 2024/07/16 05:27:40 INFO <job_856353230> Handled error whilst updating typescript: private_source_timed_out {:source=>"nexus.xxxxxxx.com/repository/npm-group"}
2024-07-16T05:27:40.3463980Z updater | 2024/07/16 05:27:40 INFO <job_856353230> Nothing to update for Dependency Group: 'typescript'

Smallest manifest that reproduces the issue

No response

abdulapopoola commented 1 month ago

@sachin-sandhu , could this be related to this PR?

sachin-sandhu commented 1 month ago

Hi @mbiggs-gresham , Can you please provide more context on how we can reproduce the issue. thanks!

mbiggs-gresham commented 1 month ago

@sachin-sandhu Yes we have multiple instances in different repos where we have a react project with the following configuration:

dependabot.yml

version: 2
registries:
  npm:
    type: npm-registry
    url: https://nexus.xxxxxxxxxxx.com/repository/npm-group
    username: dependabot
    password: ${{secrets.NEXUS}}

  - package-ecosystem: "npm"
    directory: "/project"
    schedule:
      interval: "daily"
      time: "05:00"
      timezone: "Europe/London"
    pull-request-branch-name:
      separator: "-"
    open-pull-requests-limit: 5
    registries:
      - npm

The react projects each have a .yarnrc.yml in them with the following:

compressionLevel: mixed

enableGlobalCache: false

nodeLinker: node-modules

npmRegistryServer: "https://nexus.xxxxxxxxxxx.com/repository/npm-group/"

yarnPath: .yarn/releases/yarn-4.3.1.cjs

And since the end of last week, everyday, these all now fail across our organisation with private_source_timed_out errors. 

2024-07-18T04:56:31.2904259Z Dependabot encountered '45' error(s) during execution, please check the logs for more details.
2024-07-18T04:56:31.2905540Z +--------------------------------------------------------------+
2024-07-18T04:56:31.2906546Z |                Dependencies failed to update                 |
2024-07-18T04:56:31.2907479Z +-----------------------------------+--------------------------+
2024-07-18T04:56:31.2909934Z | tslib                             | private_source_timed_out |
2024-07-18T04:56:31.2910882Z | typescript                        | private_source_timed_out |
2024-07-18T04:56:31.2911985Z | @types/react                      | private_source_timed_out |
2024-07-18T04:56:31.2913179Z | @types/react-dom                  | private_source_timed_out |
2024-07-18T04:56:31.2914136Z | react                             | private_source_timed_out |
2024-07-18T04:56:31.2915261Z | react-dom                         | private_source_timed_out |
2024-07-18T04:56:31.2916399Z | @emotion/babel-plugin             | private_source_timed_out |
2024-07-18T04:56:31.2917527Z | @emotion/react                    | private_source_timed_out |
2024-07-18T04:56:31.2918520Z | @emotion/styled                   | private_source_timed_out |
2024-07-18T04:56:31.2919640Z | @mui/icons-material               | private_source_timed_out |
2024-07-18T04:56:31.2921001Z | @mui/material                     | private_source_timed_out |
2024-07-18T04:56:31.2921994Z | @babel/runtime                    | private_source_timed_out |
2024-07-18T04:56:31.2923008Z | @babel/core                       | private_source_timed_out |
2024-07-18T04:56:31.2924276Z | @babel/eslint-parser              | private_source_timed_out |
2024-07-18T04:56:31.2925415Z | @babel/plugin-transform-runtime   | private_source_timed_out |
2024-07-18T04:56:31.2926619Z | @babel/preset-env                 | private_source_timed_out |
2024-07-18T04:56:31.2927692Z | @babel/preset-react               | private_source_timed_out |
2024-07-18T04:56:31.2928790Z | @babel/preset-typescript          | private_source_timed_out |
2024-07-18T04:56:31.2930014Z | @rollup/plugin-babel              | private_source_timed_out |
2024-07-18T04:56:31.2931123Z | @rollup/plugin-node-resolve       | private_source_timed_out |
2024-07-18T04:56:31.2932212Z | @rollup/plugin-typescript         | private_source_timed_out |
2024-07-18T04:56:31.2933296Z | rollup                            | private_source_timed_out |
2024-07-18T04:56:31.2934373Z | @typescript-eslint/eslint-plugin  | private_source_timed_out |
2024-07-18T04:56:31.2935669Z | @typescript-eslint/parser         | private_source_timed_out |
2024-07-18T04:56:31.2936660Z | eslint                            | private_source_timed_out |
2024-07-18T04:56:31.2937804Z | eslint-import-resolver-typescript | private_source_timed_out |
2024-07-18T04:56:31.2939074Z | eslint-plugin-cypress             | private_source_timed_out |
2024-07-18T04:56:31.2940201Z | eslint-plugin-import              | private_source_timed_out |
2024-07-18T04:56:31.2941399Z | eslint-plugin-n                   | private_source_timed_out |
2024-07-18T04:56:31.2942485Z | eslint-plugin-promise             | private_source_timed_out |
2024-07-18T04:56:31.2943843Z | eslint-plugin-react               | private_source_timed_out |
2024-07-18T04:56:31.2945045Z | eslint-plugin-react-hooks         | private_source_timed_out |
2024-07-18T04:56:31.2946150Z | @vitejs/plugin-react              | private_source_timed_out |
2024-07-18T04:56:31.2947113Z | vite                              | private_source_timed_out |
2024-07-18T04:56:31.2948272Z | vite-plugin-istanbul              | private_source_timed_out |
2024-07-18T04:56:31.2949622Z | @istanbuljs/nyc-config-typescript | private_source_timed_out |
2024-07-18T04:56:31.2950846Z | cypress-vite                      | private_source_timed_out |
2024-07-18T04:56:31.2951915Z | cypress-terminal-report           | private_source_timed_out |
2024-07-18T04:56:31.2952840Z | cypress                           | private_source_timed_out |
2024-07-18T04:56:31.2953965Z | @cypress/code-coverage            | private_source_timed_out |
2024-07-18T04:56:31.2954872Z | nyc                               | private_source_timed_out |
2024-07-18T04:56:31.2955901Z | @types/cypress__code-coverage     | private_source_timed_out |
2024-07-18T04:56:31.2957073Z | @cypress/vite-dev-server          | private_source_timed_out |
2024-07-18T04:56:31.2958021Z | rimraf                            | private_source_timed_out |
2024-07-18T04:56:31.2959268Z | cypress-fail-on-console-error     | private_source_timed_out |
2024-07-18T04:56:31.2960305Z +-----------------------------------+--------------------------+
2024-07-18T04:56:31.4363763Z Failure running container c198e587f22912a6888442671e85c020254c695a68a3e945a0bbe237df713ac5
2024-07-18T04:56:31.4727554Z Cleaned up container c198e587f22912a6888442671e85c020254c695a68a3e945a0bbe237df713ac5
2024-07-18T04:56:31.4865484Z   proxy | 2024/07/18 04:56:31 45/58 calls cached (77%)
2024-07-18T04:56:31.4866706Z 2024/07/18 04:56:31 Posting metrics to remote API endpoint
2024-07-18T04:56:32.0697739Z ##[error]Dependabot encountered an error performing the update

Error: The updater encountered one or more errors.

For more information see: https://github.com/xxxxxxxxxxx/project/network/updates/857361508 (write access to the repository is required to view the log)
2024-07-18T04:56:32.0710905Z 🤖 ~ finished: error reported to Dependabot ~
2024-07-18T04:56:32.0807763Z Post job cleanup.
2024-07-18T04:56:32.2147003Z Cleaning up orphan processes

One thing I have noticed is that security updates appear unaffected and still process fine.

sachin-sandhu commented 1 month ago

@mbiggs-gresham , we are looking into the issue (might request access to affected repo to be able to see the full logs). Can you in meanwhile, please use timeout in your .yarnrc.yml file just to rule out if this is not related with default timeout been overridden somehow.

httpTimeout: 240000

Edit: Can you also please provide full job log of update

mbiggs-gresham commented 1 month ago

I added the httpTimeout to the .yarnrc.yml file and then triggered dependabot again via the github website. It didn't make any difference, this is the logfile from that run. I've had to censor it as it is from our private org repo.

Separate issue but we used to be able to see the dependabot logfile from Insights -> Dependancy Graph -> Dependabot, but now we have to click the link from that page to take us to the github action and then view the log.

dependabot.txt

mbiggs-gresham commented 1 month ago

I don't know if this is related, but I went back through all of the jobs listed under Insights -> Dependency Graph -> Dependabot. All of the jobs that succeeded over 2 weeks ago show logfile directly inline on that page. All of the jobs that fail take me to a github actions page.

Our private npm repo has the 3 dependabot ip addresses whitelisted at our firewall from https://api.github.com/meta

  "dependabot": [
    "18.213.123.130/32",
    "3.217.79.163/32",
    "3.217.93.44/32"
  ],

I then found this blog https://github.blog/changelog/2024-07-10-dependabot-migration-to-github-actions-for-enterprise-cloud-and-free-pro-and-teams-accounts-with-actions-enabled/

I then found this page https://docs.github.com/en/code-security/dependabot/working-with-dependabot/configuring-access-to-private-registries-for-dependabot#configuring-firewall-ip-rules which states If you run Dependabot on GitHub Actions self-hosted runners, you should instead use the IP addresses under the actions key.. We're not using self hosted runners, but it does seem like a change made by github is now making them run as actions.

So despite the blog post stating No additional steps are required, and you should not experience service disruptions during the migration is it the case that we now have to whitelist over 4000 ip addresses just to run dependabot? Surely dependabot actions should be tying themselves to the same 3 ip addresses as standard.

EDIT: This page here https://docs.github.com/en/code-security/dependabot/working-with-dependabot/about-dependabot-on-github-actions-runners#enabling-or-disabling-dependabot-on-github-hosted-runners pretty much states that the IP addresses will change once on github hosted runners (we never requested this, we were migrated automatically). Which also implies we now have to pay for our own larger resource class or run our own self-hosted runner. Dependabot should fix this their end to tie it to the original 3 ip addresses imho.

abdulapopoola commented 1 month ago

@mbiggs-gresham sorry about that, it appears this might be the cause of that.

Yes, we recently migrated repos over and it should be seamless.

Also tagging @honeyankit and @carlincherry as FYI.

honeyankit commented 1 month ago

@mbiggs-gresham : Sorry about the issue. I request you to follow the below docs and disable Dependabot on actions from the org or repo level till the time, we fix this issue. Request you to lets us know if the disabling of Dependabot on Actions solved this particular problem.

https://docs.github.com/en/code-security/dependabot/working-with-dependabot/about-dependabot-on-github-actions-runners#enabling-or-d[…]osted-runners

Again sorry about the pain it caused due to our migration.

mbiggs-gresham commented 1 month ago

Thanks everyone, I disabled Dependabot on Actions for one of our mono repos, triggered dependabot and can confirm that PR's immediately started flooding in as normal (no other changes made).

@abdulapopoola @honeyankit Thanks, if someone could let us know please when this has been resolved, I can test it out. I presume you are going to tie the actions in to the original 3 ip addresses?

I think for us the particular pain point is that the infrastructure on our Enterprise GH account was changed underneath us with no communication. We didn't receive any prior notice, and we didn't receive any confirmation it had been migrated. This has led to 2 weeks worth of confusion.

honeyankit commented 1 month ago

@mbiggs-gresham : We completely understand the challenges you're facing, and we want to assure you that we're here to help you every step of the way. Please continue to monitor the GitHub blog for any new announcements. Due to the scale of the migration, we were unable to inform each customer individually, so we used the blog to communicate broadly. Rest assured, your organization has been added to our list of those experiencing issues during the migration, and we will reach out to you directly with updates as soon as we have a solution.

If everything is working for now, then please go ahead and close this issue. Thank you.

mbiggs-gresham commented 1 month ago

@honeyankit Without labouring the point, I think that's the problem though, enterprise organisations do not sit there monitoring the github blog for changes. Even if they did, that was blogged on the 11th July and we were migrated on the ~12th July (possibly even the 11th). It would have been far better if you had done what you had done in other cases and put a banner or message at the top of our github repo websites or at least made it opt-in.

I'd quite like to leave the issue open tbh, until it's fixed. Otherwise it may get forgotten about and still be a problem when we're eventually forced to migrate to Dependabot on Actions. We rely on dependabot quite heavily.

honeyankit commented 1 month ago

@mbiggs-gresham : Thank you for your inputs. I liked the idea of the banner or message. Sure you can keep the issue open.