Support for private repositories for pub

sigurdm commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Feature description

When enabling dependabot support for a pub project there should be a way to configure the credentials for accessing private third-party repositories.

From discussion here: https://github.com/orgs/community/discussions/14200#discussioncomment-4050762

The authentication pub uses is documented here: https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md#authentication

Basically dependabot needs to run dart pub token add <repository-url> and give it a credential token on stdin for each private repository.

It will then pass that token to the repository in a 'Authorization: Bearer ' header when doing version listings and downloading packages.

sigurdm commented 1 year ago

Re mirroring:

A third party repository at can be set up to mirror the official repository pub.dev.

In that case one would like to run the resolution with the environment variable $PUB_HOSTED_URL set to , that will then be used as the default url for fetching packages.

sigurdm commented 1 year ago

cc @jonasfj

lazytesting commented 1 year ago

Hi I would love to see this functionality. Currently dependabot fails to run on a dart/flutter project which has any 'private' dependencies as they cannot be resolved.

Relevant log lines:

  proxy | 2023/09/14 07:26:20 [555] 401 https://xxx.xxxx.xx:443/xxx-test-org/test-repo/api/packages/providerpackage
  proxy | 2023/09/14 07:26:20 [555] {"error": {"code": "UNAUTHORIZED", "message": "Invalid authentication provided."}}

updater | 2023/09/14 07:26:24 ERROR <job_721850595> Error processing providerpackage (NoMethodError)
updater | 2023/09/14 07:26:24 ERROR <job_721850595> undefined method `match?' for nil:NilClass
updater | 
updater |         version_string.match?(/^[0-9a-f]{6,}$/)
updater |                       ^^^^^^^
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/pub/lib/dependabot/pub/update_checker.rb:112:in `git_revision?'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/pub/lib/dependabot/pub/update_checker.rb:97:in `version_unless_ignored'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/pub/lib/dependabot/pub/update_checker.rb:15:in `latest_version'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:166:in `all_versions_ignored?'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:79:in `check_and_create_pull_request'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:60:in `check_and_create_pr_with_error_handling'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:35:in `block in perform'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:35:in `each'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:35:in `perform'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:64:in `run'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/update_files_command.rb:39:in `perform_job'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:53:in `run'
updater | 2023/09/14 07:26:24 ERROR <job_721850595> bin/update_files.rb:24:in `<main>'
updater | 2023/09/14 07:26:24 INFO <job_721850595> Finished job processing
updater | 2023/09/14 07:26:24 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +---------------------------------+
updater | |  Dependencies failed to update  |
updater | +-----------------+---------------+
updater | | providerpackage | unknown_error |
updater | +-----------------+---------------+

sigurdm commented 1 year ago

@mctofu just a small ping. I believe this issue contains the information you requested in https://github.com/orgs/community/discussions/14200#discussioncomment-4050762 .

What are next steps if we want to implement this?

mctofu commented 1 year ago

@sigurdm Yes, that is the info we needed. For this to work on Dependabot running on github.com our team will need to implement:

Adding support/docs for pub registries to dependabot.yml
Adding support for pub registries to our authentication proxy. This will attach the Authorization: Bearer header to outgoing requests to the pub registry without making the credentials available to the code in this repo.

Additionally, it would be helpful for us to get access to a private pub repository for testing or at least some assistance in verifying the added support works.

At the moment we don't have this on our roadmap but I'll surface this internally as a feature that is blocked on us.

lazytesting commented 1 year ago

hi @mctofu do maybe already have some news on this one? I'm happy to contribute if that would help but as far as I could see the proxy seems to be a private project.

mctofu commented 11 months ago

I added support for private Pub repos to our proxy but I don't have access to a private pub repo so I haven't tested it. It's currently available to try via the Dependabot CLI and I'm hoping someone can validate it works.

To try it out, first install the CLI.

Create a job.yml file as follows. Replace the source repo with your own as well as the hosted-url and token for the pub credentials.

---
job:
  package-manager: pub
  allowed-updates:
  - dependency-type: direct
    update-type: all
  source:
    provider: github
    repo: dependabot-fixtures/pub_with_git_dependency
    directory: "/."
    branch: main
    api-endpoint: https://api.github.com/
    hostname: github.com
credentials:
  - type: pub_repository
    hosted-url: https://example-private-pub-repo.dev/optional-path
    token: not-a-real-token

If the source repo is private, set an env variable named LOCAL_GITHUB_ACCESS_TOKEN to a github PAT which can read the repo.

Run dependabot update -f job.yml and verify you can get updates to the private dependency.

This is not yet available to configure via dependabot.yml but that would be the next step if we can confirm this works.

sigurdm commented 11 months ago

Sorry for never coming back about a private repository to test against.

We don't currently have good facilities for hosting this for you, but there are a number of third_party repository providers: https://dart.dev/tools/pub/custom-package-repositories. I know at least https://onepub.dev/ lets you create a private repository for free pretty easily.

sigurdm commented 11 months ago

@bsutton runs https://onepub.dev/, he might be able to assist.

bsutton commented 11 months ago

@mctofu the free version of OnePub should give you want but if you need anything extra happy to a donate some extra licences to the cause. Reach out if you have any issues.

lazytesting commented 11 months ago

I ran a test on my test project (https://github.com/lazytesting/flutterpackagetest/tree/main)

with this config:

job:
  package-manager: pub
  allowed-updates:
    - dependency-type: direct
      update-type: all
  source:
    provider: github
    repo: lazytesting/flutterpackagetest
    directory: /consumertest
    branch: main
    api-endpoint: https://api.github.com/
    hostname: github.com

credentials:
  - type: pub_repository
    hosted-url: https://dart.cloudsmith.io/maarten-test-org/test-repo/
    token: xxx

And it works, it both 'resolves' public packages as well as my private package which is hosted at CloudSmith

log:

updater | +-----------------------------------------------------+
updater | |         Changes to Dependabot Pull Requests         |
updater | +---------+-------------------------------------------+
updater | | created | cupertino_icons ( from 1.0.5 to 1.0.6 )   |
updater | | created | flutter_lints ( from 2.0.2 to 3.0.1 )     |
updater | | created | providerpackage ( from 1.0.0+2 to 1.2.1 ) | <==== this is a private package
updater | +---------+-------------------------------------------+ 
  proxy | 2023/12/09 16:28:29 130/396 calls cached (32%)

lazytesting commented 10 months ago

@mctofu is there anything else I can help with to get this one merged?

mctofu commented 10 months ago

@lazytesting This isn't officially supported yet and is subject to change but if you want to test this out you can configure it via dependabot.yml like:

version: 2
registries:
  my-pub-registry:
    type: pub-repository
    url: https://example-private-pub-repo.dev/optional-path
    token: ${{secrets.MY_PUB_TOKEN}}
updates:
  - package-ecosystem: "pub" # See documentation for possible values
    directory: "/" # Location of package manifests
    schedule:
      interval: "weekly"
    registries:
      - my-pub-registry

Note that hosted-url has been renamed to url.

mctofu commented 9 months ago

Two other enhancements we could consider on the dependabot-core side before closing this out:

When we fail to access a package because of missing or incorrect credentials, it'd be helpful to raise a Dependabot::PrivateSourceAuthenticationFailure so we can surface this to the user in a friendly manner (example)
I saw it's possible to override the default package repository. If we wanted to support this in Dependabot, one approach we've been taking is to look for a registry configured with the replaces-base flag set (example). If found, it could be used to set the PUB_HOSTED_URL env var when checking for updates?

@sigurdm What do you think? The PrivateSourceAuthenticationFailure would be helpful to guide users but I'm not sure how common it is to change the default repository?

sigurdm commented 9 months ago

When we fail to access a package because of missing or incorrect credentials, it'd be helpful to raise a Dependabot::PrivateSourceAuthenticationFailure

We should be able to do that. When we have a failed authentication we output either "authentication failed"(401) or "authorization failed" (403):

https://github.com/dart-lang/pub/blob/23edb4522177ecff180225228e6c1323ca58967d/lib/src/source/hosted.dart#L1616

I'm not sure how common it is to change the default repository?

I don't think it is common numerically. But I have heard of large companies that replicate the parts of pub.dev they need on a third-party host.

Also people in countries with restricted internet access might override to use a local mirror: https://docs.flutter.dev/community/china#configure-your-machine-to-use-a-mirror-site

genevievelanzer commented 9 months ago

Is it possible to ignore the private repo in the meantime? I have a dependency to a private gitlab repo in my project, and it's the only private dependency in the entire pubspec.yaml. Since auth for this is not yet supported, I was hoping to just skip it while checking for updates.

I see here that this isn't recommended, and I can confirm that trying to ignore it in my dependabot.yml does not work. Any other ideas on how I could do this ? Thanks!

sigurdm commented 9 months ago

Is it possible to ignore the private repo in the meantime?

No - darts dependabot integration only works if you can do a full resolution.

juandiago commented 1 month ago

Is this not supported already? I'm getting mixed signals here & here

dependabot / dependabot-core

Support for private repositories for pub #6121

Is there an existing issue for this?

Feature description