search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.7k stars 1.02k forks source link

Dependabot fetching incorrect url for terraform private registry modules #5707

Closed krishna-pp closed 2 years ago

krishna-pp commented 2 years ago

Is there an existing issue for this?

Package ecosystem

terraform

Package manager version

N/A

Language version

No response

Manifest location and content before the Dependabot update

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4"
    }
  }
}

module "aws_legacy_vars" {
  source = "<ourcustominstance>.jfrog.io/terraform-modules-local__<namespace>/aws-legacy-account-variables/aws"
  version = "v0.3.9"
}

output "test_eu-west-1_vpc-id" {
  value = module.aws_legacy_vars.vpc_id["test"]["eu-west-1"]
}

dependabot.yml content

---
version: 2
registries:
  terraform-modules:
    type: terraform-registry
    url: https://ourcustominstance.jfrog.io
    token: ${{ secrets.JFROG_TERRAFORM_TOKEN }}

updates:
  - package-ecosystem: "terraform"
    directory: "terraform/"
    registries:
      - terraform-modules
    schedule:
      interval: "daily"

Updated dependency

aws-legacy-account-variables from v0.3.9 to v0.3.10

What you expected to see, versus what you actually saw

Expected Dependabot to open a PR with the updates to the terraform module aws-legacy-account-variables.

Instead, it throws a RuntimeError saying it can't fetch the URL: https://ourcustominstance.jfrog.io:443/artifactory/api/terraform/v1/terraform-modules-local__namespace/aws-legacy-account-variables/aws/versions.

However, the correct URL to look for is: https://ourcustominstance.jfrog.io:443/artifactory/api/terraform/v1/modules/terraform-modules-local__namespace/aws-legacy-account-variables/aws/versions

(it is missing /modules/ after v1/). I've also posted the contents of .well-known/terraform.json, where terraform registry is hosted).

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

Please see below the dependabot logs. I'm also attaching the content of https://ourcustominstance.jfrog.io/.well-known/terraform.json down below.

proxy | time="2022-09-13T11:17:25Z" level=info msg="proxy starting" commit=b031647dc5f52d8120800fc16337727989cb9be0
  proxy | 2022/09/13 11:17:25 Listening (:1080)
updater | 2022-09-13T11:17:26.194122921 [anonymous-instance:main:WARN:src/firecracker/src/main.rs:370] You are using a deprecated parameter: --seccomp-level 2, that will be removed in a future version.
updater | 2022-09-13T11:17:26.248176971 [457966917:main:WARN:src/devices/src/legacy/serial.rs:432] Detached the serial input due to peer close/error.
updater | time="2022-09-13T11:17:29Z" level=info msg="guest starting" commit=d97478b458e198f9b9a6cb546d902ee2e6651286
updater | time="2022-09-13T11:17:29Z" level=info msg="starting job..." fetcher_timeout=5m0s job_id=457966917 updater_timeout=45m0s updater_version=8b7894be54c6b357e90d1be860cf1bbfbe3d4ea5
updater | I, [2022-09-13T11:17:33.151925 #7]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
updater | INFO <job_457966917> Starting job processing
  proxy | 2022/09/13 11:17:38 [002] GET https://api.github.com:443/repos/iZettle/tf-modules-releases-poc
  proxy | 2022/09/13 11:17:38 [002] * authenticating github api request
  proxy | 2022/09/13 11:17:38 [002] 200 https://api.github.com:443/repos/iZettle/tf-modules-releases-poc
  proxy | 2022/09/13 11:17:38 [004] GET https://api.github.com:443/repos/iZettle/tf-modules-releases-poc/git/refs/heads/main
  proxy | 2022/09/13 11:17:38 [004] * authenticating github api request
  proxy | 2022/09/13 11:17:38 [004] 200 https://api.github.com:443/repos/iZettle/tf-modules-releases-poc/git/refs/heads/main
  proxy | 2022/09/13 11:17:39 [007] GET https://github.com:443/iZettle/tf-modules-releases-poc/info/refs?service=git-upload-pack
  proxy | 2022/09/13 11:17:39 [007] * authenticating git server request (host: github.com)
  proxy | 2022/09/13 11:17:39 [007] 200 https://github.com:443/iZettle/tf-modules-releases-poc/info/refs?service=git-upload-pack
  proxy | 2022/09/13 11:17:40 [009] POST https://github.com:443/iZettle/tf-modules-releases-poc/git-upload-pack
  proxy | 2022/09/13 11:17:40 [009] * authenticating git server request (host: github.com)
  proxy | 2022/09/13 11:17:40 [009] 200 https://github.com:443/iZettle/tf-modules-releases-poc/git-upload-pack
  proxy | 2022/09/13 11:17:40 [011] POST https://github.com:443/iZettle/tf-modules-releases-poc/git-upload-pack
  proxy | 2022/09/13 11:17:40 [011] * authenticating git server request (host: github.com)
  proxy | 2022/09/13 11:17:40 [011] 200 https://github.com:443/iZettle/tf-modules-releases-poc/git-upload-pack
updater | INFO <job_457966917> Finished job processing
updater | time="2022-09-13T11:17:40Z" level=info msg="task complete" container_id=job-457966917-file-fetcher exit_code=0 job_id=457966917 step=fetcher
updater | I, [2022-09-13T11:17:42.747106 #7]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
updater | INFO <job_457966917> Starting job processing
updater | INFO <job_457966917> Starting update job for iZettle/tf-modules-releases-poc
updater | INFO <job_457966917> Checking if hashicorp/aws 4.30.0 needs updating
  proxy | 2022/09/13 11:17:46 [015] GET https://registry.terraform.io:443/.well-known/terraform.json
  proxy | 2022/09/13 11:17:46 [015] 200 https://registry.terraform.io:443/.well-known/terraform.json
  proxy | 2022/09/13 11:17:46 [017] GET https://registry.terraform.io:443/v1/providers/hashicorp/aws/versions
  proxy | 2022/09/13 11:17:46 [017] 200 https://registry.terraform.io:443/v1/providers/hashicorp/aws/versions
updater | INFO <job_457966917> Latest version is 4.30.0
updater | INFO <job_457966917> No update needed for hashicorp/aws 4.30.0
updater | INFO <job_457966917> Checking if terraform-modules-local__<namespace>/aws-legacy-account-variables/aws  needs updating
  proxy | 2022/09/13 11:17:46 [019] GET https://<ourcustominstance>.jfrog.io:443/.well-known/terraform.json
  proxy | 2022/09/13 11:17:46 [019] * authenticating terraform registry request (host: <ourcustominstance>.jfrog.io)
  proxy | 2022/09/13 11:17:47 [019] 200 https://<ourcustominstance>.jfrog.io:443/.well-known/terraform.json
  proxy | 2022/09/13 11:17:47 [021] GET https://<ourcustominstance>.jfrog.io:443/artifactory/api/terraform/v1/terraform-modules-local__<namespace>/aws-legacy-account-variables/aws/versions
  proxy | 2022/09/13 11:17:47 [021] * authenticating terraform registry request (host: <ourcustominstance>.jfrog.io)
  proxy | 2022/09/13 11:17:47 [021] 404 https://<ourcustominstance>.jfrog.io:443/artifactory/api/terraform/v1/terraform-modules-local__<namespace>/aws-legacy-account-variables/aws/versions
updater | I, [2022-09-13T11:17:47.301987 #7]  INFO -- sentry: ** [Raven] Sending event 80362a62888e4d75b633ba0a0c09a5f2 to Sentry
  proxy | 2022/09/13 11:17:47 [023] POST https://sentry.io:443/api/1451818/store/
  proxy | 2022/09/13 11:17:47 [023] 200 https://sentry.io:443/api/1451818/store/
updater | ERROR <job_457966917> Error processing terraform-modules-local__<namespace>/aws-legacy-account-variables/aws (Dependabot::DependabotError)
updater | ERROR <job_457966917> Response from registry was 404
updater | ERROR <job_457966917> /home/dependabot/terraform/lib/dependabot/terraform/registry_client.rb:177:in `http_get!'
updater | ERROR <job_457966917> /home/dependabot/terraform/lib/dependabot/terraform/registry_client.rb:80:in `all_module_versions'
updater | ERROR <job_457966917> /home/dependabot/terraform/lib/dependabot/terraform/update_checker.rb:81:in `all_module_versions'
updater | ERROR <job_457966917> /home/dependabot/terraform/lib/dependabot/terraform/update_checker.rb:72:in `latest_version_for_registry_dependency'
updater | ERROR <job_457966917> /home/dependabot/terraform/lib/dependabot/terraform/update_checker.rb:18:in `latest_version'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:517:in `all_versions_ignored?'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:216:in `check_and_create_pull_request'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:102:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `block in run'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `each'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:73:in `run'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_457966917> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_457966917> bin/update_files.rb:22:in `<main>'
updater | INFO <job_457966917> Finished job processing
updater | INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | time="2022-09-13T11:17:50Z" level=info msg="task complete" container_id=job-457966917-updater exit_code=0 job_id=457966917 step=updater

Content of https://ourcustominstance.jfrog.io/.well-known/terraform.json below:

{
  "modules.v1" : "https://ourcustominstance.jfrog.io/artifactory/api/terraform/v1/modules",
  "state.v2" : "https://ourcustominstance.jfrog.io/artifactory/api/terraform/remote/v2",
  "tfe.v2" : "https://ourcustominstance.jfrog.io/artifactory/api/terraform/remote/v2",
  "tfe.v2.1" : "https://ourcustominstance.jfrog.io/artifactory/api/terraform/remote/v2",
  "tfe.v2.2" : "https://ourcustominstance.jfrog.io/artifactory/api/terraform/remote/v2",
  "login.v1" : {
    "client" : "terraform-cli",
    "authz" : "https://ourcustominstance.jfrog.io/ui/terraform/oauth2/authorize",
    "token" : "https://ourcustominstance.jfrog.io/artifactory/api/oauth2/token",
    "grant_types" : [ "authz_code" ]
  }
}

Smallest manifest that reproduces the issue

No response

jeffwidman commented 2 years ago

👋 Thanks for taking the time to make your bug report detailed.

Code related to fetching terraform private registry URLs changed recently in https://github.com/dependabot/dependabot-core/pull/5366, it may be worth perusing that as a starter to understand how Dependabot tries to fetch from private terraform registries.

I poked through our Terraform registry client , but I'm not super familiar with Terraform , so it was difficult for me to follow.

Unfortunately this isn't something we'll have the cycles to debug on our side anytime soon, especially since there's a small chance it may end up being something specific to artifactory...

Can you try running the dry-run script to see if you can reproduce locally?

Because if so, it should be fairly straightforward to track down where it's creating the malformed URL (just throw in some puts statements 😄 ). And from there I suspect the fix would be pretty straightforward based on what I see in the registry client code.

Let me know if you need any further pointers on debugging or putting together a PR.

krishna-pp commented 2 years ago

Thanks, @jeffwidman, for the tips. I was able to reproduce the issue locally. It turns out that the service discovery URL from our private registry is missing a slash at the end.

It is documented in the terraform registry API document here: https://www.terraform.io/registry/api-docs.

The service identifier for this protocol is modules.v1, and the declared URL should always end with a slash 
such that the paths shown in the following sections can be appended to it.

I'll take it up with our registry provider. Thanks again for your help. I'm closing the issue.

jeffwidman commented 2 years ago

Glad you figured it out. 🎉

Thanks for circling back and letting us know. 👍