Credential issues when using custom source provider for bitbucket server

[noorul]

Is there an existing issue for this?

• [X] I have searched the existing issues

I made changes to dependabot-core to support the bitbucket server source.

Everything is working fine. I started using dependabot cli to verify certain things a few days back and everything was working fine even without specifying the credentials, for example, input

input:
    job:
      package-manager: maven
      allowed-updates:
        - update-type: all
      existing-pull-requests:
        - - dependency-name: com.arangodb:arangodb-java-driver
            dependency-version: 7.1.0
      source:
        provider: bitbucket_server
        repo: proj/test-repo
        directory: /
        commit: 0103c642c39289b0e0bece5494a485e5d859d5c8
      ignore-conditions:
        - dependency-name: com.arangodb:arangodb-java-driver
          version-requirement: "7.0.0"
    credentials:
      - type: git_source
         host: example.com
         token: secret
      - type: maven_repository
        url: https://xxxx.jfrog.io/xxxx/libs-release-local
        username: $JFROG_USERNAME
        password: $JFROG_PASSWORD
      - type: maven_repository
        url: https://xxxx.jfrog.io/xxxx/libs-snapshot-local
        username: $JFROG_USERNAME
        password: $JFROG_PASSWORD

But all of a sudden it stopped working. Now I get the following error:

    cli | 2023/08/03 05:52:23 Inserting $LOCAL_GITHUB_ACCESS_TOKEN into credentials
    cli | 2023/08/03 05:52:23 Adding missing credentials-metadata into job definition
    cli | 2023/08/03 05:52:23 using image ghcr.io/github/dependabot-update-job-proxy/dependabot-update-job-proxy:latest at sha256:64a9250977fc206582758ae46861428e144abf6daf74448bd2b195706bc301a0
    cli | 2023/08/03 05:52:23 using image ghcr.io/dependabot/dependabot-updater-maven at sha256:ba5ede6cfda51f3b2c06875644bf990d461c42e4204266066f8ea119b4fa370b
  proxy | 2023/08/03 05:52:24 proxy starting, commit: 7a5d8c20c9a94f571abb6857bf47b26103757412
  proxy | 2023/08/03 05:52:24 initializing metrics client: No address passed and autodetection from environment failed
  proxy | 2023/08/03 05:52:24 Listening (:1080)
updater | Updating certificates in /etc/ssl/certs...
updater | rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
updater | 1 added, 0 removed; done.
updater | Running hooks in /etc/ca-certificates/update.d...
updater | done.
updater | 2023/08/03 05:52:26 INFO Raven 3.1.2 configured not to capture errors: DSN not set
updater | 2023/08/03 05:52:27 INFO Starting job processing
  proxy | 2023/08/03 05:52:27 [002] GET https://example.com:443/rest/api/1.0/projects/proj/repos/test-repo/raw/pom.xml?at=0103c642c39289b0e0bece5494a485e5d859d5c8
  proxy | 2023/08/03 05:52:28 [002] 401 https://example.com:443/rest/api/1.0/projects/proj/repos/test-repo/raw/pom.xml?at=0103c642c39289b0e0bece5494a485e5d859d5c8
updater | 2023/08/03 05:52:28 ERROR Error during file fetching; aborting
updater | 2023/08/03 05:52:28 ERROR Dependabot::Clients::BitbucketServer::Unauthorized
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/clients/bitbucket_server.rb:261:in `get'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/clients/bitbucket_server.rb:73:in `fetch_file_contents'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/file_fetchers/base.rb:550:in `_fetch_file_content_fully_specified'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/file_fetchers/base.rb:525:in `_fetch_file_content'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/file_fetchers/base.rb:163:in `fetch_file_from_host'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/maven/lib/dependabot/maven/file_fetcher.rb:33:in `pom'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/maven/lib/dependabot/maven/file_fetcher.rb:25:in `fetch_files'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/common/lib/dependabot/file_fetchers/base.rb:77:in `files'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/dependabot-updater/lib/dependabot/file_fetcher_command.rb:67:in `dependency_files'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/dependabot-updater/lib/dependabot/file_fetcher_command.rb:30:in `perform_job'
updater | 2023/08/03 05:52:28 ERROR /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:52:in `run'
updater | 2023/08/03 05:52:28 ERROR bin/fetch_files.rb:23:in `<main>'
  proxy | 2023/08/03 05:52:28 [003] POST http://host.docker.internal:53131/update_jobs/cli/record_update_job_error
    cli | 2023/08/03 05:52:28 type was unexpected: expected create_pull_request got record_update_job_error
  proxy | 2023/08/03 05:52:28 [003] 200 http://host.docker.internal:53131/update_jobs/cli/record_update_job_error
  proxy | 2023/08/03 05:52:28 [004] PATCH http://host.docker.internal:53131/update_jobs/cli/mark_as_processed
    cli | 2023/08/03 05:52:28 missing expectation
  proxy | 2023/08/03 05:52:28 [004] 200 http://host.docker.internal:53131/update_jobs/cli/mark_as_processed
updater | 2023/08/03 05:52:28 INFO Finished job processing
updater | 2023/08/03 05:52:28 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +---------------+
updater | |    Errors     |
updater | +---------------+
updater | | unknown_error |
updater | +---------------+
  proxy | 2023/08/03 05:52:29 0/1 calls cached (0%)

I tried several combination of setting credentials for the type git_source but not helping, for example

input:
    job:
      package-manager: maven
      allowed-updates:
        - update-type: all
      existing-pull-requests:
        - - dependency-name: com.arangodb:arangodb-java-driver
            dependency-version: 7.1.0
      source:
        provider: bitbucket_server
        repo: proj/test-repo
        directory: /
        commit: 0103c642c39289b0e0bece5494a485e5d859d5c8
      ignore-conditions:
        - dependency-name: com.arangodb:arangodb-java-driver
          version-requirement: "7.0.0"
    credentials:
      - type: git_source
        host: example.com
        token: $BITBUCKET_TOKEN
      - type: maven_repository
        url: https://xxxx.jfrog.io/xxxx/libs-release-local
        username: $JFROG_USERNAME
        password: $JFROG_PASSWORD
      - type: maven_repository
        url: https://xxxx.jfrog.io/xxxx/libs-snapshot-local
        username: $JFROG_USERNAME
        password: $JFROG_PASSWORD

I think the proxy is not passing credentials as bearer tokens. How to force a proxy to pass a bearer token?

Is the code available in public for ghcr.io/github/dependabot-update-job-proxy/dependabot-update-job-proxy:latest ?

I created an issue,https://github.com/dependabot/dependabot-core/issues/7736, for this in the CLI project but did not get any response.

[jeffwidman]

The proxy isn't currently open source, but it's something we've talked about... The first step would be running the idea by the business/product/legal/security teams to get their perspective, and frankly we just haven't had the engineering bandwidth to even start those discussions yet.

At worst case, since you're running it through the CLI, you could try doing TCP dump to see what is being sent over the wire to BitBucket and then manually check whether that matches the creds you've got in the config file, as well as confirm those creds are legit if you run the API calls yourself outside of :dependabot: .

[noorul]

@jeffwidman I already intercepted the request and found that it uses username and password fields to form basic auth. But our bitbucket server does not support basic auth instead it expects a bearer token. I tried token field but it is ignored. Is there a way to tell the proxy that it has to use Bearer auth instead of Basic and use token from a field? I wonder how this works for api.github.com.

[jeffwidman]

This will probably require further investigation from our side. I don't control where that will fit in the planning/prioritization, but I will make sure those who do see this.

What version of BitBucket server are you using? And is this a privately hosted BitBucket server or BitBucket cloud?

[noorul]

This is the Bitbucket Server v8.4.1 for which I implemented various interfaces in dependabot-core.

[noorul]

@jeffwidman This is kind of blocking me from progressing. Any help is appreciated.

1. Is there a point for me to wait for any changes to proxy or should I go and run my own proxy?
2. Is it possible to bye-pass proxy and make dependabot-core updater to use credentials?