search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.66k stars 1.01k forks source link

[npm/yarn] error with auth free private npm registry #1943

Open MarcAurel opened 4 years ago

MarcAurel commented 4 years ago

I am trying to use Dependabot with Docker to maintain some JS repositories on a self-hosted Github Enterprise. I use the official dependabot/dependabot-core Docker image and a modified update-script.rb with my Github and Github Enterprise tokens.

The JS projects I want to maintain use a private npm registry (lets say verdaccio.mycompany.org) that does not need authentication at all. Here is the configuration I use in update-script.rb:

credentials =
  [
    {
    "type" => "git_source",
    "host" => "github.com",
    "username" => "x-access-token",
    "password" => "GITHUB_TOKEN"
  },
  {

    "type" => "git_source",
    "host" => "github.mycompany.org"
    "username" => "x-access-token",
    "password" => "GHE_TOKEN"
  },
  {
    "type" => "npm_registry",
    "registry" => "verdaccio.mycompany.org",
  }
]

The command I run within my Docker container is the following:

bundle install -j 3 --path vendor && bundle exec ruby ./update-script.rb

And here is the error I get:

Traceback (most recent call last):
        14: from ./update-script.rb:95:in `<main>'
        13: from ./update-script.rb:95:in `each'
        12: from ./update-script.rb:105:in `block in <main>'
        11: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-common-0.113.28/lib/dependabot/update_checkers/base.rb:29:in `up_to_date?'
        10: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-common-0.113.28/lib/dependabot/update_checkers/base.rb:246:in `requirements_up_to_date?'
         9: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker.rb:63:in `updated_requirements'
         8: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-common-0.113.28/lib/dependabot/update_checkers/base.rb:70:in `preferred_resolvable_version'
         7: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker.rb:27:in `latest_resolvable_version'
         6: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker.rb:22:in `latest_version'
         5: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker.rb:197:in `latest_version_details'
         4: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker.rb:170:in `latest_released_version'
         3: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:35:in `latest_version_from_registry'
         2: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:101:in `valid_npm_details?'
         1: from /home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:273:in `npm_details'
/home/dependabot/dependabot-script/vendor/ruby/2.6.0/gems/dependabot-npm_and_yarn-0.113.28/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:319:in `check_npm_response':
The following source could not be reached as it requires authentication (and any provided details were invalid
or lacked the required permissions): verdaccio.mycompany.org (Dependabot::PrivateSourceAuthenticationFailure)

I have no difficulties to reach my private registry from the Docker container: curl https://verdaccio.mycompany.org/react/ gives me similar results as https://registry.npmjs.org/react/.

Could you enlighten me on this? I am not a ruby ​​developer, but I would be more than happy to contribute at my ability level if I have the opportunity.

feelepxyz commented 4 years ago

@MarcAurel could you try removing the entry for verdaccio in credentials? Think it should work without this if it's public. Adding it to creds might be confusing dependabot to try and add auth to the request.

MarcAurel commented 4 years ago

@feelepxyz thank you for the quick answer! I get the exact same error without the entry for verdaccio in the settings... 😕

feelepxyz commented 4 years ago

@MarcAurel could you log out the npm_response from this method when requesting from verdaccio to see if there's something unexpected there? https://github.com/dependabot/dependabot-core/blob/main/npm_and_yarn/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb#L357

MarcAurel commented 4 years ago

I finally managed to continue investigating this issue (thank you for your patience @feelepxyz ☺️).

This is not an authentication problem: some requests to verdaccio.mycompany.org have a 404 status because some dependencies of our project are not published on our private npm registry. With the help of Yarn workspaces, these dependencies are located directly in our project.

Is this usecase covered by Dependabot?

yeikel commented 3 years ago

Did you fix this issue?

I am experiencing the same with the following configuration : 

  {
    "type" => "npm_registry",
    "url" => "https://myRegistry/nodejs/content/groups/npm/"
  }

This is the same URL I have in npm locally (npm config list)

➜  ~ npm config list
; "user" config from /Users/yvaldess/.npmrc

registry = "https://myRegistry/nodejs/content/groups/npm/" 

; node bin location = /Users/yeikel/.nvm/versions/node/v16.1.0/bin/node
; cwd = /Users/yeikel
; HOME = /Users/yeikel
; Run `npm config ls -l` to show all defaults.
➜  ~ npm config list
; "user" config from /Users/yeikel/.npmrc

registry = "https://myRegistry/nodejs/content/groups/npm/" 

; node bin location = /Users/yvaldess/.nvm/versions/node/v16.1.0/bin/node
; cwd = /Users/yeikel
; HOME = /Users/yeikel
; Run `npm config ls -l` to show all defaults

    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:114:in `block in locked_registry'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:114:in `each'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:114:in `find'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:114:in `locked_registry'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:32:in `registry'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:82:in `registry_url'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb:40:in `dependency_url'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:375:in `dependency_url'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:304:in `fetch_npm_response'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:285:in `npm_details'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:104:in `valid_npm_details?'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:35:in `latest_version_from_registry'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker.rb:180:in `latest_released_version'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker.rb:205:in `latest_version_details'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-npm_and_yarn-0.149.3/lib/dependabot/npm_and_yarn/update_checker.rb:23:in `latest_version'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-common-0.149.3/lib/dependabot/update_checkers/base.rb:231:in `numeric_version_up_to_date?'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-common-0.149.3/lib/dependabot/update_checkers/base.rb:189:in `version_up_to_date?'
    /Users/yeikel/.rvm/gems/ruby-2.6.6/gems/dependabot-common-0.149.3/lib/dependabot/update_checkers/base.rb:33:in `up_to_date?'

It seems to boil down to this : 

def registry
          locked_registry || first_registry_with_dependency_details
end

@feelepxyz Could you please help here?

Edit :

It ended up being a typo in my configuration. The configuration format is not very consistent.

For maven it is type and url but for NPM it is type and registry

Is this intentional? 

credentials = [
  {
    "type" => "maven_repository",
    "url" => "https://url"
  },
  {
    "type" => "npm_registry",
    "registry" => "url"
  }
]

Edit 2 :

I had to remove HTTPS from my configuration because this https://github.com/dependabot/dependabot-core/blob/ecfabf6e7c1622b7a958df7f521cd2b533ca237f/npm_and_yarn/lib/dependabot/npm_and_yarn/update_checker/registry_finder.rb#L82 adds the protocol again and the final URL looked like https://https:// That looks like a bug to me

One again, I did not see this behaviour in maven

ShayanBits commented 1 year ago

I get the same error when accessing some public registries like the following example: Any solutions for this case?

- Updating @vue/test-utils (from 1.3.5)…/home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:326:in `check_npm_response': The following source could not be reached as it requires authentication (and any provided details were invalid or lacked the required permissions): registry.npmjs.org (Dependabot::PrivateSourceAuthenticationFailure)
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:279:in `npm_details'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:108:in `valid_npm_details?'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker/latest_version_finder.rb:36:in `latest_version_from_registry'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker.rb:180:in `latest_released_version'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker.rb:205:in `latest_version_details'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-npm_and_yarn-0.172.2/lib/dependabot/npm_and_yarn/update_checker.rb:23:in `latest_version'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-common-0.172.2/lib/dependabot/update_checkers/base.rb:231:in `numeric_version_up_to_date?'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-common-0.172.2/lib/dependabot/update_checkers/base.rb:189:in `version_up_to_date?'
    from /home/dependabot/dependabot-script/.bundle/gems/dependabot-common-0.172.2/lib/dependabot/update_checkers/base.rb:33:in `up_to_date?'
    from ./generic-update-script.rb:179:in `block in <main>'
    from ./generic-update-script.rb:169:in `each'
    from ./generic-update-script.rb:169:in `<main>'