search

tinglesoftware / dependabot-azure-devops

Tools for updating dependencies in Azure DevOps repositories using https://dependabot.com
MIT License
181 stars 59 forks source link

`AZURE_PORT` and `AZURE_VIRTUAL_DIRECTORY` are ignored in the repository URLs #1034

Open frazar opened 3 months ago

frazar commented 3 months ago

Describe the bug I have a repository that uses both pip and npm to manage the dependencies of the different software modules therein. The repository is hosted on an Azure DevOps configured with a non-default port (8443) and a /tfs virtual directory, so the repository URL looks like this:

https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name

When running the ghcr.io/tinglesoftware/dependabot-updater:latest docker image to target the pip dependencies, everything works as expected.

Instead, when targeting the npm dependencies, the following exception is encountered:

Cloning repository into /home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name
/home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:102:in `rescue in clone_repo_contents': Dependabot::RepoNotFound (Dependabot::RepoNotFound)
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:92:in `clone_repo_contents'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:in `clone_repo_contents'
        from bin/update-script.rb:523:in `<main>'
/home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:345:in `run_shell_command': Cloning into '/home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name'... (Dependabot::SharedHelpers::HelperSubprocessFailed)
fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443: Connection refused
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:610:in `block in _clone_repo_contents'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:187:in `with_git_configured'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:591:in `_clone_repo_contents'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:94:in `clone_repo_contents'
        from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:in `clone_repo_contents'
        from bin/update-script.rb:523:in `<main>'

It appears that the repo URL used by the npm_and_yarn module is missing the Azure port (8443) and Azure virtual directory (/tfs) specified in the respective env vars. To confirm this suspicion, I manually corrected the repo URL as follows:

-https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/
+https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/

and then paste the URL into a browser. Indeed, I can confirm that the 1st URL does not work, whereas the 2nd one does.

Also note that the URL is computed correctly when setting DEPENDABOT_PACKAGE_MANAGER='pip'.

To Reproduce Steps to reproduce the behavior:

  1. Set up an Azure DevOps server instance with port 8443 and virtual directory "tfs"
  2. Run the following script to update both the pip and npm dependencies:
#!/usr/bin/env bash

set -ueo pipefail
set -x

DOCKER_IMAGE_TAG='ghcr.io/tinglesoftware/dependabot-updater:latest'
GITHUB_ACCESS_TOKEN='REDACTED'
AZURE_HOSTNAME='tfs.domain.com'
AZURE_PORT='8443'
AZURE_VIRTUAL_DIRECTORY='tfs'
AZURE_ORGANIZATION='Azure_Org_Name'
AZURE_ACCESS_TOKEN='REDACTED'
AZURE_PROJECT='Azure_Project_Name'
AZURE_REPOSITORY='Azure_Repo_Name'

docker pull "$DOCKER_IMAGE_TAG"

# Update pip dependencies: SUCCESS
DEPENDABOT_DIRECTORY='/path/to/python/module'
DEPENDABOT_PACKAGE_MANAGER='pip'
docker run --rm -i \
    --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \
    --env AZURE_HOSTNAME="$AZURE_HOSTNAME" \
    --env AZURE_VIRTUAL_DIRECTORY="$AZURE_VIRTUAL_DIRECTORY" \
    --env AZURE_PORT="$AZURE_PORT" \
    --env AZURE_ORGANIZATION="$AZURE_ORGANIZATION" \
    --env AZURE_PROJECT="$AZURE_PROJECT" \
    --env AZURE_REPOSITORY="$AZURE_REPOSITORY" \
    --env AZURE_ACCESS_TOKEN="$AZURE_ACCESS_TOKEN" \
    --env DEPENDABOT_PACKAGE_MANAGER="$DEPENDABOT_PACKAGE_MANAGER" \
    --env DEPENDABOT_DIRECTORY="$DEPENDABOT_DIRECTORY" \
    --env DEPENDABOT_SKIP_PULL_REQUESTS="true" \
    "$DOCKER_IMAGE_TAG"

# Update npm dependencies: FAILS
DEPENDABOT_DIRECTORY='/path/to/javascript/module'
DEPENDABOT_PACKAGE_MANAGER='npm_and_yarn'
docker run --rm -i \
    --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \
    --env AZURE_HOSTNAME="$AZURE_HOSTNAME" \
    --env AZURE_VIRTUAL_DIRECTORY="$AZURE_VIRTUAL_DIRECTORY" \
    --env AZURE_PORT="$AZURE_PORT" \
    --env AZURE_ORGANIZATION="$AZURE_ORGANIZATION" \
    --env AZURE_PROJECT="$AZURE_PROJECT" \
    --env AZURE_REPOSITORY="$AZURE_REPOSITORY" \
    --env AZURE_ACCESS_TOKEN="$AZURE_ACCESS_TOKEN" \
    --env DEPENDABOT_PACKAGE_MANAGER="$DEPENDABOT_PACKAGE_MANAGER" \
    --env DEPENDABOT_DIRECTORY="$DEPENDABOT_DIRECTORY" \
    --env DEPENDABOT_SKIP_PULL_REQUESTS="true" \
    "$DOCKER_IMAGE_TAG"
  1. Notice that dependabot succeeds for pip, but fails for npm

Expected behavior The run for the npm should correctly build the repo URL based on the AZURE_PORT and AZURE_VIRTUAL_DIRECTORY environment variables.

Screenshots

Full logs when running the script above: ```bash $ ./run.reproduce.sh + DOCKER_IMAGE_TAG=ghcr.io/tinglesoftware/dependabot-updater:latest + GITHUB_ACCESS_TOKEN=REDACTED + AZURE_HOSTNAME=tfs.domain.com + AZURE_PORT=8443 + AZURE_VIRTUAL_DIRECTORY=tfs + AZURE_ORGANIZATION=Azure_Org_Name + AZURE_ACCESS_TOKEN=REDACTED + AZURE_PROJECT=Azure_Project_Name + AZURE_REPOSITORY=Azure_Repo_Name + docker pull ghcr.io/tinglesoftware/dependabot-updater:latest latest: Pulling from tinglesoftware/dependabot-updater Digest: sha256:78fc62ca084076f5f1d01b364521717f7eac7f3088d0a3ec05205794ae51aa51 Status: Image is up to date for ghcr.io/tinglesoftware/dependabot-updater:latest ghcr.io/tinglesoftware/dependabot-updater:latest + DEPENDABOT_DIRECTORY=/path/to/python/module + DEPENDABOT_PACKAGE_MANAGER=pip + docker run --rm -i --env GITHUB_ACCESS_TOKEN=REDACTED --env AZURE_HOSTNAME=tfs.domain.com --env AZURE_VIRTUAL_DIRECTORY=tfs --env AZURE_PORT=8443 --env AZURE_ORGANIZATION=Azure_Org_Name --env AZURE_PROJECT=Azure_Project_Name --env AZURE_REPOSITORY=Azure_Repo_Name --env AZURE_ACCESS_TOKEN=REDACTED --env DEPENDABOT_PACKAGE_MANAGER=pip --env DEPENDABOT_DIRECTORY=/path/to/python/module --env DEPENDABOT_SKIP_PULL_REQUESTS=true ghcr.io/tinglesoftware/dependabot-updater:latest warning: parser/current is loading parser/ruby31, which recognizes 3.1.4-compliant syntax, but you are running 3.1.3. Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. GitHub access token has been provided. Pull requests limit is set to zero. Security only updates are implied. Using 'https://tfs.domain.com:8443/tfs/' as API endpoint Working in Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name, 'default' branch under '/path/to/python/module' directory Looking for configuration file in the repository ... 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 404 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yaml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 404 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yaml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 Configuration file was not found, a default config will be used. 😔 Fetching pip dependency files ... 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/3228eccf3aa79b0afc90ca382c3e609f6824b4b8?recursive=false 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/3228eccf3aa79b0afc90ca382c3e609f6824b4b8?recursive=false 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/pyproject.toml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/pyproject.toml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/198b514db6bad6a6afd3dc9852c8e35911cb9eea?recursive=false 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/198b514db6bad6a6afd3dc9852c8e35911cb9eea?recursive=false 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/dev-requirements.in&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/dev-requirements.in&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/dev-requirements.txt&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/dev-requirements.txt&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/requirements.txt&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/requirements/requirements.txt&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/src&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/src&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/b9b919fa7edd1a644cfa11e45a948b35c94ab3a7?recursive=false 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/trees/b9b919fa7edd1a644cfa11e45a948b35c94ab3a7?recursive=false 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/setup.py&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/setup.py&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/.python-version&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=path/to/python/module/.python-version&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 Found 6 dependency file(s) at commit 5163809403b1a11ad00b78381e5eb99d62bdc679 - /path/to/python/module/pyproject.toml - /path/to/python/module/requirements/dev-requirements.in - /path/to/python/module/requirements/dev-requirements.txt - /path/to/python/module/requirements/requirements.txt - /path/to/python/module/setup.py - /path/to/python/module/.python-version Parsing dependencies information Found 3 dependencies - coverage (7.4.3) - pytest (8.0.1) - pytest-randomly (3.15.0) 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/_apis/connectionData 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/_apis/connectionData 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/pullrequests?api-version=6.0&searchCriteria.status=active&searchCriteria.creatorId=8c74abf8-44d6-46da-a890-a42537db0deb&searchCriteria.targetRefName=refs/heads/main 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/pullrequests?api-version=6.0&searchCriteria.status=active&searchCriteria.creatorId=8c74abf8-44d6-46da-a890-a42537db0deb&searchCriteria.targetRefName=refs/heads/main Checking if coverage 7.4.3 is vulnerable coverage 7.4.3 is not vulnerable Checking if pytest 8.0.1 is vulnerable pytest 8.0.1 is not vulnerable Checking if pytest-randomly 3.15.0 is vulnerable pytest-randomly 3.15.0 is not vulnerable Done + DEPENDABOT_DIRECTORY=/path/to/javascript/module + DEPENDABOT_PACKAGE_MANAGER=npm_and_yarn + docker run --rm -i --env GITHUB_ACCESS_TOKEN=REDACTED --env AZURE_HOSTNAME=tfs.domain.com --env AZURE_VIRTUAL_DIRECTORY=tfs --env AZURE_PORT=8443 --env AZURE_ORGANIZATION=Azure_Org_Name --env AZURE_PROJECT=Azure_Project_Name --env AZURE_REPOSITORY=Azure_Repo_Name --env AZURE_ACCESS_TOKEN=REDACTED --env DEPENDABOT_PACKAGE_MANAGER=npm_and_yarn --env DEPENDABOT_DIRECTORY=/path/to/javascript/module --env DEPENDABOT_SKIP_PULL_REQUESTS=true ghcr.io/tinglesoftware/dependabot-updater:latest warning: parser/current is loading parser/ruby31, which recognizes 3.1.4-compliant syntax, but you are running 3.1.3. Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri. GitHub access token has been provided. Pull requests limit is set to zero. Security only updates are implied. Using 'https://tfs.domain.com:8443/tfs/' as API endpoint Working in Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name, 'default' branch under '/path/to/javascript/module' directory Looking for configuration file in the repository ... 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 <-- 200 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/stats/branches?name=main 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 404 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 --> GET https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yaml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 🌍 <-- 404 https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_apis/git/repositories/Azure_Repo_Name/items?path=.github/dependabot.yaml&versionDescriptor.versionType=commit&versionDescriptor.version=5163809403b1a11ad00b78381e5eb99d62bdc679 Configuration file was not found, a default config will be used. 😔 Cloning repository into /home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:102:in `rescue in clone_repo_contents': Dependabot::RepoNotFound (Dependabot::RepoNotFound) from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:92:in `clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:in `clone_repo_contents' from bin/update-script.rb:523:in `
' /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:345:in `run_shell_command': Cloning into '/home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name'... (Dependabot::SharedHelpers::HelperSubprocessFailed) fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443: Connection refused from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:610:in `block in _clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:187:in `with_git_configured' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:591:in `_clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:94:in `clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:in `clone_repo_contents' from bin/update-script.rb:523:in `
' ```
mburumaxwell commented 3 months ago

The dependabot-updater image is deprecated and hasn't been updated for over a year (https://github.com/orgs/tinglesoftware/packages?repo_name=dependabot-azure-devops).

Retry this with ecosystem specific docker images and report back.

Also, it may be easier to try the extension first (it should work with Azure DevOps server)

frazar commented 3 months ago

Thank you very much for your prompt answer!

The dependabot-updater image is deprecated and hasn't been updated for over a year (https://github.com/orgs/tinglesoftware/packages?repo_name=dependabot-azure-devops).

Sorry for that. Indeed, I was following an outdated walkthrough article.

Retry this with ecosystem specific docker images and report back.

I tried with the following Docker images:

but got a similar errors, this time for both pip and npm.

fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443 after 2593 ms: Connection refused

I can see the URL is again missing the port and virtual directory

-https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/
+https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/
Script for reproducing ```bash #!/usr/bin/env bash set -ueo pipefail set -x GITHUB_ACCESS_TOKEN='REDACTED' AZURE_HOSTNAME='tfs.domain.com' AZURE_PORT='8443' AZURE_VIRTUAL_DIRECTORY='tfs' AZURE_ORGANIZATION='Azure_Org_Name' AZURE_ACCESS_TOKEN='REDACTED' AZURE_PROJECT='Azure_Project_Name' AZURE_REPOSITORY='Azure_Repo_Name' DEPENDABOT_TARGET_BRANCH='main' DEPENDABOT_AUTHOR_NAME='dependabot' DEPENDABOT_AUTHOR_EMAIL='dependabot@domain.com' DEPENDABOT_DIRECTORY='/path/to/python/module' DEPENDABOT_PACKAGE_MANAGER='pip' DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=1 DEPENDABOT_LABELS='[]' DOCKER_IMAGE_TAG="ghcr.io/tinglesoftware/dependabot-updater-$DEPENDABOT_PACKAGE_MANAGER:latest" docker pull "$DOCKER_IMAGE_TAG" docker run --rm -i \ --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \ --env AZURE_HOSTNAME="$AZURE_HOSTNAME" \ --env AZURE_VIRTUAL_DIRECTORY="$AZURE_VIRTUAL_DIRECTORY" \ --env AZURE_PORT="$AZURE_PORT" \ --env AZURE_ORGANIZATION="$AZURE_ORGANIZATION" \ --env AZURE_PROJECT="$AZURE_PROJECT" \ --env AZURE_REPOSITORY="$AZURE_REPOSITORY" \ --env AZURE_ACCESS_TOKEN="$AZURE_ACCESS_TOKEN" \ --env DEPENDABOT_AUTHOR_EMAIL="$DEPENDABOT_AUTHOR_EMAIL" \ --env DEPENDABOT_AUTHOR_NAME="$DEPENDABOT_AUTHOR_NAME" \ --env DEPENDABOT_TARGET_BRANCH="$DEPENDABOT_TARGET_BRANCH" \ --env DEPENDABOT_PACKAGE_MANAGER="$DEPENDABOT_PACKAGE_MANAGER" \ --env DEPENDABOT_DIRECTORY="$DEPENDABOT_DIRECTORY" \ --env DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT="$DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT" \ --env DEPENDABOT_LABELS="$DEPENDABOT_LABELS" \ --env DEPENDABOT_SKIP_PULL_REQUESTS="true" \ "$DOCKER_IMAGE_TAG" \ update_script ```
Full output logs ``` + GITHUB_ACCESS_TOKEN=REDACTED + AZURE_HOSTNAME=tfs.domain.com + AZURE_PORT=8443 + AZURE_VIRTUAL_DIRECTORY=tfs + AZURE_ORGANIZATION=Azure_Org_Name + AZURE_ACCESS_TOKEN=REDACTED + AZURE_PROJECT=Azure_Project_Name + AZURE_REPOSITORY=Azure_Repo_Name + DEPENDABOT_TARGET_BRANCH=main + DEPENDABOT_AUTHOR_NAME=dependabot + DEPENDABOT_AUTHOR_EMAIL=dependabot@domain.com + DEPENDABOT_DIRECTORY=/path/to/python/module + DEPENDABOT_PACKAGE_MANAGER=pip + DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=1 + DEPENDABOT_LABELS='[]' + DOCKER_IMAGE_TAG=ghcr.io/tinglesoftware/dependabot-updater-pip:latest + docker pull ghcr.io/tinglesoftware/dependabot-updater-pip:latest latest: Pulling from tinglesoftware/dependabot-updater-pip Digest: sha256:70b2061f718043a00dc094d252e78998c92f556fbad1eb63736b73b609527011 Status: Image is up to date for ghcr.io/tinglesoftware/dependabot-updater-pip:latest ghcr.io/tinglesoftware/dependabot-updater-pip:latest + docker run --rm -i --env GITHUB_ACCESS_TOKEN=REDACTED --env AZURE_HOSTNAME=tfs.domain.com --env AZURE_VIRTUAL_DIRECTORY=tfs --env AZURE_PORT=8443 --env AZURE_ORGANIZATION=Azure_Org_Name --env AZURE_PROJECT=Azure_Project_Name --env AZURE_REPOSITORY=Azure_Repo_Name --env AZURE_ACCESS_TOKEN=REDACTED --env DEPENDABOT_AUTHOR_EMAIL=dependabot@domain.com --env DEPENDABOT_AUTHOR_NAME=dependabot --env DEPENDABOT_TARGET_BRANCH=main --env DEPENDABOT_PACKAGE_MANAGER=pip --env DEPENDABOT_DIRECTORY=/path/to/python/module --env DEPENDABOT_OPEN_PULL_REQUESTS_LIMIT=1 --env 'DEPENDABOT_LABELS=[]' --env DEPENDABOT_SKIP_PULL_REQUESTS=true ghcr.io/tinglesoftware/dependabot-updater-pip:latest update_script GitHub access token has been provided. Using 'https://tfs.domain.com:8443/tfs/' as API endpoint Pull Requests shall be linked to milestone (work item) 0 Pull Requests shall be labeled [] Working in Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name, 'main' branch under '/path/to/python/module' directory Cloning repository into /home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/file_fetchers/base.rb:175:in `rescue in clone_repo_contents': Cloning into '/home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name'... (Dependabot::RepoNotFound) fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443 after 2593 ms: Connection refused from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/file_fetchers/base.rb:163:in `clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `bind_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `validate_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added' from bin/update_script.rb:505:in `
' /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/shared_helpers.rb:429:in `run_shell_command': Cloning into '/home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name'... (Dependabot::SharedHelpers::HelperSubprocessFailed) fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443 after 2593 ms: Connection refused from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:167:in `bind_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:167:in `validate_call_skip_block_type' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:109:in `block in create_validator_slow_skip_block_type' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/file_fetchers/base.rb:792:in `block in _clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/shared_helpers.rb:265:in `with_git_configured' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `bind_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `validate_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/file_fetchers/base.rb:776:in `_clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `bind_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `validate_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/dependabot-common-0.247.0/lib/dependabot/file_fetchers/base.rb:165:in `clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `bind_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/call_validation.rb:270:in `validate_call' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/gems/sorbet-runtime-0.5.11294/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added' from bin/update_script.rb:505:in `
' ```
EtienneBelanger commented 3 months ago

Any update on this ? I have the same error on a nuget package manager setup using the extention

fatal: repository 'https://tfs.orga.com/Orga/MozOrga/_git/ApiAffaires/' not found

The Url should be : https://tfs.orga.com:443/tfs/Orga/MozOrga/_git/ApiAffaires/

Thulasi225 commented 2 months ago

Describe the bug I have a repository that uses both pip and npm to manage the dependencies of the different software modules therein. The repository is hosted on an Azure DevOps configured with a non-default port (8443) and a /tfs virtual directory, so the repository URL looks like this:

https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name When running the ghcr.io/tinglesoftware/dependabot-updater:latest docker image to target the pip dependencies, everything works as expected.

Instead, when targeting the npm dependencies, the following exception is encountered:

Cloning repository into /home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:102:in rescue in clone_repo_contents': Dependabot::RepoNotFound (Dependabot::RepoNotFound) from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:92:inclone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:in clone_repo_contents' from bin/update-script.rb:523:in

' /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:345:in run_shell_command': Cloning into '/home/dependabot/dependabot-updater/tmp/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name'... (Dependabot::SharedHelpers::HelperSubprocessFailed) fatal: unable to access 'https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/': Failed to connect to tfs.domain.com port 443: Connection refused from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:610:inblock in _clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/shared_helpers.rb:187:in with_git_configured' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:591:in_clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/common/lib/dependabot/file_fetchers/base.rb:94:in clone_repo_contents' from /home/dependabot/dependabot-updater/vendor/ruby/3.1.0/bundler/gems/dependabot-core-8919de6bed26/npm_and_yarn/lib/dependabot/npm_and_yarn/file_fetcher.rb:41:inclone_repo_contents' from bin/update-script.rb:523:in `
' It appears that the repo URL used by the npm_and_yarn module is missing the Azure port (8443) and Azure virtual directory (/tfs) specified in the respective env vars. To confirm this suspicion, I manually corrected the repo URL as follows:

-https://tfs.domain.com/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/ +https://tfs.domain.com:8443/tfs/Azure_Org_Name/Azure_Project_Name/_git/Azure_Repo_Name/ and then paste the URL into a browser. Indeed, I can confirm that the 1st URL does not work, whereas the 2nd one does.

Also note that the URL is computed correctly when setting DEPENDABOT_PACKAGE_MANAGER='pip'.

To Reproduce Steps to reproduce the behavior:

Set up an Azure DevOps server instance with port 8443 and virtual directory "tfs" Run the following script to update both the pip and npm dependencies:

!/usr/bin/env bash

set -ueo pipefail set -x

DOCKER_IMAGE_TAG='ghcr.io/tinglesoftware/dependabot-updater:latest' GITHUB_ACCESS_TOKEN='REDACTED' AZURE_HOSTNAME='tfs.domain.com' AZURE_PORT='8443' AZURE_VIRTUAL_DIRECTORY='tfs' AZURE_ORGANIZATION='Azure_Org_Name' AZURE_ACCESS_TOKEN='REDACTED' AZURE_PROJECT='Azure_Project_Name' AZURE_REPOSITORY='Azure_Repo_Name'

docker pull "$DOCKER_IMAGE_TAG"

Update pip dependencies: SUCCESS

DEPENDABOT_DIRECTORY='/path/to/python/module' DEPENDABOT_PACKAGE_MANAGER='pip' docker run --rm -i \ --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \ --env AZURE_HOSTNAME="$AZURE_HOSTNAME" \ --env AZURE_VIRTUAL_DIRECTORY="$AZURE_VIRTUAL_DIRECTORY" \ --env AZURE_PORT="$AZURE_PORT" \ --env AZURE_ORGANIZATION="$AZURE_ORGANIZATION" \ --env AZURE_PROJECT="$AZURE_PROJECT" \ --env AZURE_REPOSITORY="$AZURE_REPOSITORY" \ --env AZURE_ACCESS_TOKEN="$AZURE_ACCESS_TOKEN" \ --env DEPENDABOT_PACKAGE_MANAGER="$DEPENDABOT_PACKAGE_MANAGER" \ --env DEPENDABOT_DIRECTORY="$DEPENDABOT_DIRECTORY" \ --env DEPENDABOT_SKIP_PULL_REQUESTS="true" \ "$DOCKER_IMAGE_TAG"

Update npm dependencies: FAILS

DEPENDABOT_DIRECTORY='/path/to/javascript/module' DEPENDABOT_PACKAGE_MANAGER='npm_and_yarn' docker run --rm -i \ --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" \ --env AZURE_HOSTNAME="$AZURE_HOSTNAME" \ --env AZURE_VIRTUAL_DIRECTORY="$AZURE_VIRTUAL_DIRECTORY" \ --env AZURE_PORT="$AZURE_PORT" \ --env AZURE_ORGANIZATION="$AZURE_ORGANIZATION" \ --env AZURE_PROJECT="$AZURE_PROJECT" \ --env AZURE_REPOSITORY="$AZURE_REPOSITORY" \ --env AZURE_ACCESS_TOKEN="$AZURE_ACCESS_TOKEN" \ --env DEPENDABOT_PACKAGE_MANAGER="$DEPENDABOT_PACKAGE_MANAGER" \ --env DEPENDABOT_DIRECTORY="$DEPENDABOT_DIRECTORY" \ --env DEPENDABOT_SKIP_PULL_REQUESTS="true" \ "$DOCKER_IMAGE_TAG" Notice that dependabot succeeds for pip, but fails for npm Expected behavior The run for the npm should correctly build the repo URL based on the AZURE_PORT and AZURE_VIRTUAL_DIRECTORY environment variables.

Screenshots

Full logs when running the script above:

mburumaxwell commented 2 months ago

PR fix for this is welcome

mburumaxwell commented 3 weeks ago

Quick update to everyone tracking this issue. We do not use Azure DevOps Server or any earlier OnPrem version of it. This means we cannot fix and test this. It will have to be a community contribution.

The areas to look at: