terraform-provider-azuredevop network(firewall) access for terraform init command

[dduleep]

What are the network access requirements for terraform-provider-azuredevop? I have allowed outbout to https://github.com/microsoft/terraform-provider-azuredevops/* However, I am encountering the following error.

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding microsoft/azuredevops versions matching ">= 0.1.0"...
╷
│ Error: Failed to install provider
│ 
│ Error while installing microsoft/azuredevops v1.4.0: could not query
│ provider registry for registry.terraform.io/microsoft/azuredevops: failed
│ to retrieve authentication checksums for provider: the request failed after
│ 2 attempts, please try again later: Get
│ "[https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/73f3fbba-801c-4b97-8231-a860db17091c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241027T061511Z&X-Amz-Expires=300&X-Amz-Signature=f15772077a27a523a563bc5159cf6d86c95274569aaf0f709abcb50025cc4b38&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.4.0_SHA256SUMS&response-content-type=application%2Foctet-stream"](https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/73f3fbba-801c-4b97-8231-a860db17091c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241027%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241027T061511Z&X-Amz-Expires=300&X-Amz-Signature=f15772077a27a523a563bc5159cf6d86c95274569aaf0f709abcb50025cc4b38&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.4.0_SHA256SUMS&response-content-type=application%2Foctet-stream%22):
│ EOF
╵

[xuzhang3]

terraform init pulls the provider from terraform registry not github release

[dduleep]

Before execution the terraform init on the self hosted agent(inside the private network), terraform execution environment requires network connectivity to download the terraform-provider-azuredevops provider

I have a self-hosted agent behind the secure network. I couldn't find any document related to network access to for terraform-provider-azuredevops. as per my understand following, URL needs to be whitelisted

https://github.com/microsoft/terraform-provider-azuredevops/*
https://objects.githubusercontent.com/github-production-release-asset-2e65be/*

[xuzhang3]

@dduleep add export TF_LOG=TRACE to your env and run terraeform init you will see where Terraform download provider from Some of the URLs: https://releases.hashicorp.com/terraform-provider-azuredevops/

[dduleep]

@xuzhang3 I have enabled the debug but I couldn't see any requeset for https://releases.hashicorp.com/terraform-provider-azuredevops/

2024-11-14T13:36:28.685+0300 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2024-11-14T13:36:28.685+0300 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T13:36:28.685+0300 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2024-11-14T13:36:28.685+0300 [DEBUG] GET https://registry.terraform.io/v1/providers/microsoft/azuredevops/1.2.0/download/linux/386
2024-11-14T13:36:28.685+0300 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/microsoft/azuredevops/1.2.0/download/linux/386
2024-11-14T13:36:28.768+0300 [DEBUG] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS
2024-11-14T13:36:28.768+0300 [TRACE] HTTP client GET request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS
2024-11-14T13:36:29.221+0300 [TRACE] HTTP client GET request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream
2024-11-14T13:36:29.607+0300 [ERROR] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS request failed: Get "[https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream"](https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream%22): EOF
2024-11-14T13:36:29.607+0300 [DEBUG] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS: retrying in 1s (1 left)
2024-11-14T13:36:30.607+0300 [INFO]  Previous request to the remote registry failed, attempting retry.
2024-11-14T13:36:30.607+0300 [TRACE] HTTP client GET request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS
2024-11-14T13:36:30.647+0300 [TRACE] HTTP client GET request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream
2024-11-14T13:36:31.021+0300 [ERROR] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS request failed: Get "[https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream"](https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream%22): EOF
╷
│ Error: Failed to install provider
│ 
│ Error while installing microsoft/azuredevops v1.2.0: could not query
│ provider registry for registry.terraform.io/microsoft/azuredevops: failed
│ to retrieve authentication checksums for provider: the request failed after
│ 2 attempts, please try again later: Get
│ "[https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream"](https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241114T103629Z&X-Amz-Expires=300&X-Amz-Signature=48658e26dfd3f1ce1c9307ec5d66d6b09971a51b5a1add20c32fadfb8d1350cb&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream%22):
│ EOF
╵

[xuzhang3]

@dduleep This is not the default terraform registry configuration.

[dduleep]

@xuzhang3 I have used following provider configuration as per the documentation

terraform {
  required_providers {
    azuredevops = {
      source = "microsoft/azuredevops"
      version = "=1.2.0"
    }
  }
}

[xuzhang3]

microsoft/azuredevops will download the provider from the Terraform registry not GH release. You should have changed some of your local configurations. terraform.rc?

[dduleep]

@xuzhang3 I don't have terraform.rc or terrafomrc files. I have tried windows environment also, but it also downloaded from GitHub.

complete debug log in windows 11 environment

terraform init
[WARN] Invalid log level: "_DEBUG_". Defaulting to level: TRACE. Valid levels are: [TRACE DEBUG INFO WARN ERROR OFF][WARN] Invalid log level: "_DEBUG_". Defaulting to level: TRACE. Valid levels are: [TRACE DEBUG INFO WARN ERROR OFF]2024-11-18T08:26:36.093+0300 [DEBUG] Adding temp file log sink: D:\Users\xxxx\AppData\Local\Temp\terraform-log278573571
2024-11-18T08:26:36.096+0300 [INFO]  Terraform version: 1.0.11
2024-11-18T08:26:36.097+0300 [INFO]  Go runtime version: go1.16.4
2024-11-18T08:26:36.097+0300 [INFO]  CLI args: []string{"D:\\Users\\xxxx\\Projects\\Terraform\\terraform.exe", "init"}
2024-11-18T08:26:36.097+0300 [TRACE] Stdout is a terminal of width 232
2024-11-18T08:26:36.097+0300 [TRACE] Stderr is a terminal of width 232
2024-11-18T08:26:36.097+0300 [TRACE] Stdin is a terminal
2024-11-18T08:26:36.108+0300 [DEBUG] Attempting to open CLI config file: D:\Users\xxxx\AppData\Roaming\terraform.rc
2024-11-18T08:26:36.108+0300 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2024-11-18T08:26:36.108+0300 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2024-11-18T08:26:36.109+0300 [DEBUG] ignoring non-existing provider search directory D:\Users\xxxx\AppData\Roaming\terraform.d\plugins
2024-11-18T08:26:36.111+0300 [DEBUG] ignoring non-existing provider search directory D:\Users\xxxx\AppData\Roaming\HashiCorp\Terraform\plugins
2024-11-18T08:26:36.111+0300 [INFO]  CLI command args: []string{"init"}

Initializing the backend...
2024-11-18T08:26:36.350+0300 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2024-11-18T08:26:36.350+0300 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2024-11-18T08:26:36.350+0300 [DEBUG] New state was assigned lineage "292cdd21-096d-6022-33b5-dccb4211dba1"
2024-11-18T08:26:36.350+0300 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2024-11-18T08:26:36.350+0300 [TRACE] Meta.Backend: instantiated backend of type <nil>
2024-11-18T08:26:36.351+0300 [DEBUG] checking for provisioner in "."
2024-11-18T08:26:36.351+0300 [DEBUG] checking for provisioner in "D:\\Users\\xxxx\\Projects\\Terraform"
2024-11-18T08:26:36.428+0300 [INFO]  Failed to read plugin lock file .terraform\plugins\windows_amd64\lock.json: open .terraform\plugins\windows_amd64\lock.json: The system cannot find the path specified.

Initializing provider plugins...
2024-11-18T08:26:36.428+0300 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
- Finding microsoft/azuredevops versions matching "1.2.0"...
2024-11-18T08:26:36.440+0300 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2024-11-18T08:26:36.440+0300 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2024-11-18T08:26:36.440+0300 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2024-11-18T08:26:36.440+0300 [TRACE] statemgr.Filesystem: read nil snapshot
2024-11-18T08:26:36.440+0300 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2024-11-18T08:26:36.440+0300 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2024-11-18T08:26:37.758+0300 [DEBUG] GET https://registry.terraform.io/v1/providers/microsoft/azuredevops/versions
2024-11-18T08:26:37.758+0300 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/microsoft/azuredevops/versions
2024-11-18T08:26:38.214+0300 [TRACE] providercache.fillMetaCache: scanning directory .terraform\providers
2024-11-18T08:26:38.214+0300 [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform\providers: CreateFile .terraform: The system cannot find the file specified.
2024-11-18T08:26:38.215+0300 [TRACE] providercache.fillMetaCache: error while scanning directory .terraform\providers: cannot search .terraform\providers: CreateFile .terraform\providers: The system cannot find the path specified.  
2024-11-18T08:26:38.215+0300 [DEBUG] GET https://registry.terraform.io/v1/providers/microsoft/azuredevops/1.2.0/download/windows/amd64
2024-11-18T08:26:38.215+0300 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/microsoft/azuredevops/1.2.0/download/windows/amd64
2024-11-18T08:26:38.667+0300 [DEBUG] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS
2024-11-18T08:26:38.667+0300 [TRACE] HTTP client GET request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS
2024-11-18T08:26:39.339+0300 [TRACE] HTTP client GET request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fab7b8c6-3429-446f-9920-d98fb4d0195d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Crede
ntial=releaseassetproduction%2F20241118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241118T052639Z&X-Amz-Expires=300&X-Amz-Signature=ae1a278a5cdaab74d9cf0ce71b17e0c017cb8787580c508281594b9f75e0250a&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS&response-content-type=application%2Foctet-stream
2024-11-18T08:26:40.916+0300 [DEBUG] GET https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS.sig
2024-11-18T08:26:40.916+0300 [TRACE] HTTP client GET request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_SHA256SUMS.sig
2024-11-18T08:26:41.219+0300 [TRACE] HTTP client GET request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/9518cd1c-a012-4184-8410-324abd3b0451?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Crede
ntial=releaseassetproduction%2F20241118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241118T052641Z&X-Amz-Expires=300&X-Amz-Signature=c90b790665577dcb5b1777df2f6ba9707264df88be87f7af6a783319ee14060a&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_SHA256SUMS.sig&response-content-type=application%2Foctet-stream
- Installing microsoft/azuredevops v1.2.0...
2024-11-18T08:26:41.849+0300 [TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/microsoft/azuredevops v1.2.0 from https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_windows_amd64.zip
2024-11-18T08:26:41.849+0300 [TRACE] HTTP client GET request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_windows_amd64.zip
2024-11-18T08:26:42.180+0300 [TRACE] HTTP client GET request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/273244625/fc5074bc-dc3e-48bb-8bd7-f55ed238cad4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Crede
ntial=releaseassetproduction%2F20241118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241118T052642Z&X-Amz-Expires=300&X-Amz-Signature=16651dc7f22dd616b34022128679f1c2b5ae29054d5ddc5a6b12b150dde77275&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dterraform-provider-azuredevops_1.2.0_windows_amd64.zip&response-content-type=application%2Foctet-stream
╷
│ Error: Failed to install provider
│
│ Error while installing microsoft/azuredevops v1.2.0: unsuccessful request to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_windows_amd64.zip: 403
│ Forbidden
╵

[xuzhang3]

This is odd. Initializing the provider fails with a 403. This seems to be a network issue. Can you call the URL https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_windows_amd64.zip directly?

[dduleep]

I'm inside the secured(firwall) nework and currrenly that is blocking all download from the [github](https://objects.githubusercontent.com/). Therefore not able to access to https://github.com/microsoft/terraform-provider-azuredevops/releases/download/v1.2.0/terraform-provider-azuredevops_1.2.0_windows_amd64.zip directly directly

Can you tell me or share any docuements which urls/IPs need to be whitelisted to use terraform-provider-azuredevops