search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.58k stars 9.54k forks source link

Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers #26532

Closed snemetz closed 3 years ago

snemetz commented 4 years ago

Terraform Version

Terraform v0.13.4
Also tried 0.13.3 got same issue

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"
  required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

Expected Behavior

Expected init to work

Actual Behavior

Initializing provider plugins...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws: could not query provider registry for registry.terraform.io/hashicorp/aws: the request failed after 2 attempts, please try again later: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Steps to Reproduce

terraform init

Additional Context

I can use curl and access the url that terraform says it can't access with no issues I've tried multiple times over 2 days and issue doesn't change. Terraform always fails, curl always succeeds Had no issues with 0.12, but 0.13 has not worked yet Terraform section was the only change made going to 0.13 This is on Mac I also tried with only the posted terraform config and still get same issue

crosbymichael1 commented 4 years ago

Im having the same issue trying to use the helm provider:

terraform {
  required_providers {
    helm = {
      source = "hashicorp/helm"
      version = "1.3.2"
    }
  }
}

provider "helm" {

  kubernetes {
    host                   = local.kube_host
    cluster_ca_certificate = local.kube_cluster_ca_certificate_decoded
    token                  = local.kube_token
    load_config_file       = false
  }
}

Error Message:

`Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.

Initializing provider plugins...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/helm: no available releases match the given constraints 1.3.2 `

pselle commented 4 years ago

Thank you for the report! Short of more information to help replicate this issue, I've tagged this as a registry issue as perhaps the concern may lie there.

@crosbymichael1 -- it looks like your message cut of at the error message, so I'm not sure you have the same error here (the timeout).

crosbymichael1 commented 4 years ago

@pselle I figured out what was going on

so I did a tg apply in one of the helm folders and got this error, which I thought was pretty weird because there is 100% a

helm provider version > 1.3 Initializing provider plugins...

After changing the version a bunch of times I found out that since I already had a different version of the binary in my tf plugins folder, that instead of downloading the new version, terraform just checked locally to see what I had installed, and since only 10.5 is there, well that is the only version I'd be allowed to use /Users/michael.crosby/.terraform.d/plugins/registry.terraform.io/hashicorp/helm/10.5 So I just deleted my local helm provider folder and then did a tg apply and it downloaded the version I specified and it worked fine then

pselle commented 4 years ago

Thanks for the additional information @crosbymichael1! I've re-added the new label here because that gives more information that might help someone create a reproduction case for this particular angle of the issue (which could be separate from the original post, unsure, but what you've described does seem problematic, and tricky to detangle!)

crosbymichael1 commented 4 years ago

@snemetz Try to delete your local aws file in the terraform plugins folder and then rerun terraform init and see if it resolves your issue:

rm -rf ~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws

If your changing aws provider version I have a feeling if terraform see you already have a provider installed locally then its only checking for versions within your local folder and not out in the terraform registry.

snemetz commented 4 years ago

That dir was empty. plugin-cache had a lot of versions. Remove all ` ls -lR total 8 -rw-r--r-- 1 stevennemetz staff 394 Nov 16 2018 checkpoint_signature drwxr-xr-x 3 stevennemetz staff 96 Jul 1 2019 plugin-cache drwxr-xr-x 3 stevennemetz staff 96 Jun 21 2019 plugins

./plugin-cache: total 0 drwxr-xr-x 2 stevennemetz staff 64 Oct 14 19:37 darwin_amd64

./plugin-cache/darwin_amd64:

./plugins: total 0 drwxr-xr-x 2 stevennemetz staff 64 Nov 5 2019 darwin_amd64

./plugins/darwin_amd64:`

Made no change. Same error. The error claims it is about downloading the versions file. Which should be before comparing with local files.

snemetz commented 4 years ago

Debug log ` 2020/10/14 19:49:59 [INFO] Terraform version: 0.13.4 2020/10/14 19:49:59 [INFO] Go runtime version: go1.14.7 2020/10/14 19:49:59 [INFO] CLI args: []string{"/usr/local/bin/terraform13.4", "init"} 2020/10/14 19:49:59 [DEBUG] Attempting to open CLI config file: /Users/stevennemetz/.terraformrc 2020/10/14 19:49:59 [DEBUG] File doesn't exist, but doesn't need to. Ignoring. 2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins" 2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64" 2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins 2020/10/14 19:49:59 [DEBUG] will search for provider plugins in /Users/stevennemetz/.terraform.d/plugins 2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: /Users/stevennemetz/.terraform.d/plugins is a symlink to /Users/stevennemetz/.terraform.d/plugins 2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Users/stevennemetz/Library/Application Support/io.terraform/plugins 2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins 2020/10/14 19:49:59 [INFO] CLI command args: []string{"init"} 2020/10/14 19:49:59 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config 2020/10/14 19:49:59 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory 2020/10/14 19:49:59 [DEBUG] New state was assigned lineage "d727669d-a920-07ec-4a64-bc51e80e8402" 2020/10/14 19:49:59 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend) 2020/10/14 19:49:59 [TRACE] Meta.Backend: instantiated backend of type 2020/10/14 19:49:59 [DEBUG] checking for provisioner in "." 2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/usr/local/bin" 2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins" 2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64" 2020/10/14 19:49:59 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory 2020/10/14 19:49:59 [TRACE] Meta.Backend: backend does not support operations, so wrapping it in a local backend 2020/10/14 19:49:59 [TRACE] backend/local: state manager for workspace "default" will:

techdragon commented 4 years ago

I have this issue as well, its been happening since I switched to using any 0.13.x version. To me it looks like the new provider resolution functionality doesn't function correctly with having set a plugin_cache_dir setting.

techdragon commented 4 years ago

This may actually be a case of https://github.com/hashicorp/terraform/issues/26223

Edit: Yep, I just fixed my case of this issue with mv ~/.terraform.d/plugins ~/.terraform.d/plugin-cache and updating my .terraformrc file from plugin_cache_dir = "$HOME/.terraform.d/plugins" to plugin_cache_dir = "$HOME/.terraform.d/plugin-cache" problem solved instantly with all existing cached providers still recognised.

@snemetz Does this work for you?

C-Howard commented 4 years ago

I can't speak for snemetz but I appear to be seeing the same issue and I have no files in ~/.terraform.d/ besides a checkpoint_signature file

mildwonkey commented 4 years ago

It looks like there are two different problems being reported in this issue: a registry timeout error for the AWS provider, and some provider installer-specific questions and issues. I'll see how many I can address!

Error: Failed to query available provider packages w/net/http: request canceled while waiting for connection

Unfortunately I am not sure what caused this - I can't reproduce the issue now, and it's suspicious that curl was working for you while init was failing. I see that the same error messages were reported on two different days; do any of you still have this still problem today? I will check with our internal registry team and see if they have any ideas or information.

Terraform not updating provider version

Terraform v0.13 will not query the registry for any provider it finds locally installed under ${CONFIG_DIR}.terraform/plugins. Run terraform init -upgrade to tell terraform to query the registry for provider versions and install any updates meeting your version constraints.

This mechanism is changing in v0.14 so there is a separate installation directory for locally installed (non-registry) providers; I'm afraid I will mangle the details but you can take a look at the (draft!!) of the v0.14 upgrade guide for a nice explanation of the new options: https://github.com/hashicorp/terraform/blob/master/website/upgrade-guides/0-14.html.markdown#the-local-provider-cache-directory

The TL;DR is that we recommend you use the filesystem_mirror configuration for local providers, but in v0.14 you will also be able to drop them in the following directory (relative to the terraform configuration files): terraform.d/plugins/ . It's important that we can designate directories that only terraform is allowed to write to, and will make sure the documentation is clear and add warnings where appropriate so it's clear to y'all.

Issues when setting plugin_cache_dir to implicit local mirror directory #26223

Terraform expects to write to the plugin_cache_dir and so it is invalid as a local mirror. We'll take a look at improving the documentation and printing a warning or error message if the provider installer finds problematic configuration like this. I'd like to keep further conversation about this specifically in #26223 , please.

mildwonkey commented 4 years ago

Quick update: the registry team dug into their logs based on the timestamp on the original reported issue, and did not see anything suspicious, nor were there any reported outages with our various service providers. At this point I can't offer any explanation for the timeout beyond "vague unspecified network hiccups", but we'll definitely keep our eyes open and dig in if this happens again.

C-Howard commented 4 years ago

I am currently verifying and seeing the issue consistently. 

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 2.70.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

My versions.tf file for the module seeing the error is the following 

terraform {
  required_providers {
    aws = {
     source  = "hashicorp/aws"
     version = "~> 2.70.0"
    }
  }
  required_version = ">= 0.13"
}

This looks to be the same scenario as the original reported issue. We use no in-house providers, only the hashicorp AWS provider.

mildwonkey commented 4 years ago

@C-Howard thanks! Is that happening right now ? I'll report this to the registry folks; it will help if you can share your TRACE logs

C-Howard commented 4 years ago

Yes it happens every time I run terraform init.

[terragrunt] [[redacted]/account] 2020/10/19 12:59:11 Running command: terraform --version
[terragrunt] 2020/10/19 12:59:15 Terraform version: 0.13.4
[terragrunt] 2020/10/19 12:59:15 Reading Terragrunt config file at [redacted]/account/terragrunt.hcl
[terragrunt] 2020/10/19 12:59:23 Running command: terraform init -backend-config=bucket=[redacted] -backend-config=dynamodb_table=[redacted] -backend-config=encrypt=true -backend-config=key=[redacted] -backend-config=region=us-east-1 -upgrade
2020/10/19 12:59:23 [INFO] Terraform version: 0.13.4  
2020/10/19 12:59:23 [INFO] Go runtime version: go1.14.7
2020/10/19 12:59:23 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init", "-backend-config=bucket=[redacted]", "-backend-config=dynamodb_table=[redacted]", "-backend-config=encrypt=true", "-backend-config=key=[redacted]/terraform.tfstate", "-backend-config=region=[redacted]", "-upgrade"}
2020/10/19 12:59:23 [DEBUG] Attempting to open CLI config file: /Users/[redacted]/.terraformrc
2020/10/19 12:59:23 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/.terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/Library/Application Support/io.terraform/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins

Initializing provider plugins...
2020/10/19 13:00:16 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/10/19 13:00:16 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
- Finding hashicorp/aws versions matching "~> 2.70.0"...
2020/10/19 13:00:16 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:16 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:26 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:26 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:36 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded
2020/10/19 13:00:36 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/19 13:00:37 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/19 13:00:37 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:48 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context
deadline exceeded (Client.Timeout exceeded while awaiting headers)

[terragrunt] 2020/10/19 13:00:48 Hit multiple errors:
exit status 1
mildwonkey commented 4 years ago

Thanks, I appreciate the quick response - we'll dig into this on our end.

For anyone else reading this issue: Any new information is appreciated; anyone else experiencing this same issue please reply to the original comment with a 👍 (instead of adding comments that don't add information).

snemetz commented 4 years ago

This does not seem to be a registry issue I can run curl with the same url successfully before and after the failed terraform run. I've done this on a number of days, always with the same results

Unless this can be figured out, 0.13 is a non-starter at my company

snemetz commented 4 years ago

Not sure if this will help. But I removed ~/.terraform and the error changed. Now it displays

` Initializing provider plugins...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws: could not connect to registry.terraform.io: Failed to request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": context deadline exceeded (Client.Timeout exceeded while awaiting headers) `

And via curl curl https://registry.terraform.io/.well-known/terraform.json {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}

justincampbell commented 4 years ago

@snemetz Thanks for the additional info! We're still not able to find any indication of failing requests at our application or CDN layer. Could you try a more detailed cURL request? I don't have the exact headers Terraform sends off-hand, but something like:

curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

I'll try to recreate the exact headers and options now and update you when I have a better arg set.

Also, where are you running these commands from? Are you on a home or office network, VPN, or if you're running this within a cloud provider, which provider and region?

justincampbell commented 4 years ago

I did verify that the cURL above is the request that Terraform is making:

Accept-Encoding: gzip
Connection: keep-alive
Host: registry.terraform.io
User-Agent: Terraform/0.13.4
X-Terraform-Version: 0.13.4
snemetz commented 4 years ago 
curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

Works but terraform fails right afterward. In same shell

$ terraform13.4 init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

I'm running from home on a Mac.

mildwonkey commented 4 years ago

Well this is Officially Weird!

We do update the go version and various dependencies with pretty much every release, and it is always possible that something in there is interacting with your local setup.

Is there anything customized or atypical about your home network? Do you have a proxy/VPN/firewall/custom dns settings? What version mac os are you on?

C-Howard commented 4 years ago

As before I am not the original poster but seeing the same issue and it looks to be a definite issue with delay due to DNS. Typically due to internal server connectivity we connect to DNS servers running out of the office and eventually fall back to a major provider like 1.1.1.1. I have not been using those dns servers outside of the office but using the fallback dns server. This results in roughly two dns server timeouts before a request succeeds.

So you can reproduce this issue on mac by having dns servers similar to the following

Unreachable server 1
Unreachable server 2
1.1.1.1

This will reliably cause the observed failure, and if you rearrange servers to something like

1.1.1.1
Unreachable server 1
Unreachable server 2

You will find your requests resolving again, which is probably why you are not seeing anything on the registry side. This suggests that any noticeable delay in dns resolution will cause the timeout to be hit.

mildwonkey commented 4 years ago

@snemetz : @C-Howard 's last comment touches on what I was about to suggest - the old "It's always DNS" adage. Can you check your DNS configuration, or do you have a proxy?

I'm not as skilled at pouring through the changelogs, so I couldn't say just yet if this has to do with upgrading our golang version or some other dependency. Here are some maybe-related issues in the golang issue repository (emphasis on maybe). https://github.com/golang/go/issues/30702 https://github.com/golang/go/issues/23008 https://github.com/golang/go/issues/33006

jacobbednarz commented 4 years ago

I was also experiencing a similar issue with the cloudflare provider however we have managed to find a workaround that is showing promise(🤞). In our case, we upgraded the provider block to explicit 0.13 syntax and we hadn't yet ran an apply against all configurations. Some of the confgurations would be pulling /-/cloudflare as the /cloudflare/cloudflare version and while the plan would succeed without any changes, it wasn't until I forced a noop apply against all configurations that this went away for us.

It's worth noting, I wasn't able to replicate the issue consistently though. We would see it periodically within our automated CI tests every few hours.

kzap commented 3 years ago

I am experiencing the same thing with Terraform 0.13.4 and 0.13.5

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.12"...
2020/10/29 17:21:31 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/29 17:21:31 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/29 17:21:41 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:21:41 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:21:51 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/10/29 17:21:51 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/29 17:21:52 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/29 17:21:52 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:22:02 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

My openssl is 

$ openssl version
LibreSSL 2.6.5

Same result using openssl from brew

$ openssl version
OpenSSL 1.1.1h  22 Sep 2020

This does not happen when I am on a VPN located in AWS US-East-1, but not on my home ISP

paultyng commented 3 years ago

You can override the default registry timeout (10 seconds) by setting TF_REGISTRY_CLIENT_TIMEOUT to a number of seconds. You can also override the number of retries (default 1) by setting TF_REGISTRY_DISCOVERY_RETRY. I'm not sure what the issue is here but I wonder if possibly bumping the timeout or something could help you mitigate this, or at least give us another data point.

paultyng commented 3 years ago

Also if its possible, you may want to test the 0.14.0-beta2 release. Its built with a different Go version, which has a slightly different http.Client implementation, so it may work differently.

pradeepbhadani commented 3 years ago

I am also experiencing the same issue with Terraform 0.13 and Terraform 0.14- Issue #26716. I tried TF_REGISTRY_CLIENT_TIMEOUT and TF_REGISTRY_DISCOVERY_RETRY option but did not help.

mildwonkey commented 3 years ago

I was finally able to reproduce the timeout issue and confirm the necessary fix: https://github.com/hashicorp/terraform/issues/26716#issuecomment-720549842

The TL;DR is that it is indeed the linked, upstream golang issue: binaries build with cgo don't honor certain DNS settings on mac osx (more info in the linked comment above). Anyone with this problem and a golang development setup can test this out by building terraform with CGO_ENABLED=1 and trying again.

We are reaching out to our release team to modify terraform's build, and we'll also need to coordinate with our in-house provider development teams and communicate to third-party provider developers.

mildwonkey commented 3 years ago

update: I closed the other issue, so to save y'all a click here's the same comment that I linked to above:

This is how I reproduced this on my local mac:

  1. modified my DNS settings to use entirely non-existent DNS resolvers (3.3.3.3, 2.2.2.2)
  2. manually edited /etc/resolv.conf to use actual dns resolvers
    • curl, dig, etc registry.terraform.io: works just fine
    • terraform init: fails w/ timeout
  3. re-build terraform locally with CGO_ENABLED=1
    • now it works

So, as we've suspected for awhile, the issues lies with golang/go#12524. We have reached out to our build team here at hashicorp about re-working our build to not use CGO.

One caveat: when we make this change, terraform will no longer have this issue, but it's entirely possible that folks will still see this same problem again because individual providers might be built with CGO (terraform init would succeed but the actual plan/apply could fail with similar looking errors).

We will try and coordinate this build change with the provider developers we have direct relationships with, and communicate the necessity with all provider developers, but since we don't control their build processes we can't guarantee that all providers will make this same change. (this is not immediately relevant to the issue at hand, but I wanted to get the information out there in for future reference).

mmrobins commented 3 years ago

FWIW set homebrew to do CGO_ENABLED=1 https://github.com/Homebrew/homebrew-core/pull/65148

fawaf commented 3 years ago

still occurs on Terraform v0.13.5

0dragosh commented 3 years ago

Still having this issue on v0.14.4.

paultyng commented 3 years ago

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

0dragosh commented 3 years ago

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

Timing coincides, thanks a lot for letting me know!

LBoraz commented 3 years ago

i have the issue in 0.14.4 with the kubectl provider. For some reason it tries to download it from hashicorp/kubectl instead of gavinbunney/kubectl

Could not retrieve the list of available versions for provider hashicorp/kubectl: provider registry registry.terraform.io does not have a provider named registry.terraform.io/hashicorp/kubectl

pselle commented 3 years ago

@LBoraz Thank you for trying to connect to a relevant issue, but I think you've actually run into something else! We also have discussion forums that might help in the future.

What I suspect is happening for you is that you have a module that doesn't have the required_providers block declared. Required providers/provider source is a per-module concept, so if you have a module that doesn't have it declared, Terraform is trying to find it in the default namespace (hashicorp) and failing, thus your error.

carmine73 commented 3 years ago

The issue is still present

$ terraform version 
Terraform v0.14.8

In my case only if I use $HOME/.terraform.d/plugins as plugin_cache_dir (as @techdragon suggested)

$ cat .terraformrc 
# plugins path
# NOT working
plugin_cache_dir   = "$HOME/.terraform.d/plugins"
# working
#plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
#plugin_cache_dir   = "$HOME/.terraform.d/dummy"

this is my version.tf used just to test terraform init

$ cat versions.tf 
terraform {
  # providers
  required_providers {
    vcd = {
      source = "vmware/vcd"
      version = "3.2.0"
    }
  }
  # terraform version
  required_version = ">= 0.14"
}
MaxiPalle commented 3 years ago

I still have this issue with 0.15.0. Any updates on this?

srinavin7 commented 3 years ago

Same here. I have this issue with 0.15.0

Lucisano commented 3 years ago

Also having this issue

afry-siam commented 3 years ago

I'm still getting this issue too, MacOS 11.2.3, Terraform verison 0.15.3

Downgraded to Terraform 0.12.31, and now I can run terraform init successfully!

This problem completely borks a fresh install of Terraform. I couldn't even get copy-pasted example code from the official docs running, because I could never get past terraform init.

Seems like a cool tool, I hope this gets resolved soon! For the meantime I'm downgrading to the earlier version.

For those also seeking to downgrade Terraform Switcher has worked great.

jonstelly commented 3 years ago

It seems like this issue may have multiple underlying causes. For me, it seems to be a DNS issue.

I originally shared a log here about seeing this error in WSL2, but like others if I downgraded to terraform 0.13 things worked fine.

@apparentlymart replied there about differences between 0.13 and 0.14 but my issue went away for a couple days so I had nothing to go on. Since then, I've been 100% unable to run terraform init unless I do the following:

If I modify my /etc/resolv.conf to switch my DNS server to 8.8.8.8 (google dns) all terraform commands work as expected. If I switch it back to my standard/default nameserver that WSL assigns, I am completely unable to run any terraform commands that hit the network.

If I turn on TF_LOG I see things like:

2021-05-24T17:30:32.735-0500 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/random/versions
2021-05-24T17:30:42.737-0500 [ERROR] GET https://registry.terraform.io/v1/providers/hashicorp/random/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/random/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

While I can have a second terminal window open looping on curl https://registry.terraform.io/v1/providers/hashicorp/kubernetes/versions and I will get 0 failures.

@apparentlymart mentioned some possible TLS changes, but if this were just a TLS or SSL issue, I don't believe changing nameservers would reliably resolve the issue for me. Below are my results of dig ..., one to my default nameserver, and another to google's dns server.

Editorial: I love terraform but this really needs to get resolved. Look at the number of issues if you search for the client timeout error message. I know those issues go back a few years and terraform has a lot of users but most of those issues have a good number of comments and other people commenting that they're having the same issue. There has to be some fundamental networking issue in terraform or the http client library it's using, otherwise simultaneous curl requests would be failing just like terraform.

I'm going to find some time to try and compile and run terraform from source, if one of the developers wants to add a mess of logging output or try anything to resolve this, send me a branch name and I'll happily test it out and help as much as I can.

dig registry.terraform.io '@172.25.80.1'

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io @172.25.80.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65509
;; flags: qr rd ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  0       IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.66.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.2.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.130.49
dualstack.q2.shared.global.fastly.net. 0 IN A   151.101.194.49
a.gtld-servers.net.     0       IN      A       192.5.6.30
b.gtld-servers.net.     0       IN      A       192.33.14.30
c.gtld-servers.net.     0       IN      A       192.26.92.30
d.gtld-servers.net.     0       IN      A       192.31.80.30
e.gtld-servers.net.     0       IN      A       192.12.94.30
f.gtld-servers.net.     0       IN      A       192.35.51.30
g.gtld-servers.net.     0       IN      A       192.42.93.30
h.gtld-servers.net.     0       IN      A       192.54.112.30

;; Query time: 40 msec
;; SERVER: 172.25.80.1#53(172.25.80.1)
;; WHEN: Mon May 24 17:16:08 CDT 2021
;; MSG SIZE  rcvd: 484
dig registry.terraform.io '@8.8.8.8'

; <<>> DiG 9.16.1-Ubuntu <<>> registry.terraform.io @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15680
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;registry.terraform.io.         IN      A

;; ANSWER SECTION:
registry.terraform.io.  119     IN      CNAME   dualstack.q2.shared.global.fastly.net.
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.2.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.66.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.130.49
dualstack.q2.shared.global.fastly.net. 12 IN A  151.101.194.49

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon May 24 17:16:02 CDT 2021
;; MSG SIZE  rcvd: 165
brookemarshassoc commented 3 years ago

I have the same issue above. Resolved it by updating my DNS settings.

xcollantes commented 3 years ago

I turned on my VPN and this resolved my issue. From what I read in other posts there is an issue with the DNS resolution with this version of Terraform.

Before VPN:

After turning on VPN:

kindacoder commented 3 years ago

I have the same issue above. Resolved it by updating my DNS settings.

what change you did ?

kindacoder commented 3 years ago

https://github.com/hashicorp/terraform/issues/26532#issuecomment-854888178 what change you did @brookemarshassoc

jbardin commented 3 years ago

Hello,

Starting with Terraform v1.0.1 the MacOS binary is natively compiled, allowing Terraform to use the host resolver. This should avoid most issues with DNS configuration, many of which show up as timeouts like shown here.

Since this will likely resolve most of the issues represented by this thread I'm going to close it out. If anyone still encounters similar timeouts from the Terraform CLI on MacOS with v1.0.1 or newer, please file a new issue with the necessary information to reproduce it.

Thanks!

jonstelly commented 3 years ago

I'm still seeing this on WSL2 under windows. I'll get clean repro steps and create a new issue today or tomorrow.