search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.84k stars 9.19k forks source link

terraform apply hanging for long time #5419

Closed ghost closed 6 years ago

ghost commented 6 years ago

This issue was originally opened by @sureshoao as hashicorp/terraform#18580. It was migrated here as a result of the provider split. The original body of the issue is below.

main.tf 
provider "aws" {
  region = "us-east-1"
}
resource "aws_instance" "web" {
  ami           = "ami-b70554c8"
  instance_type = "t2.micro"

  tags {
    Name = "HelloWorld"
  }
}
terraform apply

2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: caused by: Post https://sts.amazonaws.com/: EOF
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 12
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: POST / HTTP/1.1
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Host: sts.amazonaws.com
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: User-Agent: aws-sdk-go/1.14.31 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Content-Length: 43
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=ASIAXDWZ4B6U4W3J2KH3/20180801/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=aea1a110d837b40ac3d992a22e244df33400578900fd646078b16a96e843a787
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: X-Amz-Date: 20180801T145840Z
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: X-Amz-Security-Token: FQoGZXIvYXdzECAaDFO+v5nVdl07xgcyLyKlArCjiiepIMm2I0DFZ5XCyTX0ahmssS8JLCoQThdf6oFkHev+6xhT+Ioh/4eUEG4kpXuCHyk6hA/cR7RKW2nmhyVaHMygcBRlXr0voihN1y4dV728ETJuiI8MaMIhxmsom9Y6OgiMJ5MUcZMLhx9FVv0kwmv/ufqosTtJ/kyWPweRAEBSCpvgFuL18nvZ00Qclwza6NTMFTmdzXNIpYrobsFffaG27ED6ARxPYnIGof+fW+bvOGQgDBva1dVD6mMw5D3xcUGq3+4wkQAExNAQaBa7RibaO7jW/al3VK3bQ9Y3U9FjU5a4o05BumgmOIHktf1A/WEgaIDgxxq7ELTZw/rG/TABul9BCadJRTLqqhCuf31V6GVdw+yucP4bqGj4D8tBE32hKOKMh9sF
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Accept-Encoding: gzip
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: -----------------------------------------------------
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"
sureshoao commented 6 years ago

please resolve this issue

jjkirby commented 6 years ago

I am having a similar problem with v1.30.0_x4 on a Mac 10.13.6 with go1.10.3 darwin/amd64. It appears to be a DNS issue because I can do a nslookup/dig on the sts endpoint. Is this possibly a go issue?

data.http.workstation-external-ip - *terraform.NodeRefreshableDataResourceInstance
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Building AWS region structure
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Building AWS auth structure
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Setting AWS metadata API timeout to 100ms
data.http.workstation-external-ip: Refreshing state...
2018-08-08T09:56:36.745-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018-08-08T09:56:36.751-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2018-08-08T09:56:36.751-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Initializing DeviceFarm SDK connection
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: POST / HTTP/1.1
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Host: sts.us-east-1.amazonaws.com
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: User-Agent: aws-sdk-go/1.14.33 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Content-Length: 43
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ7TUHS4VCO5WT7PQ/20180808/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=22bda1b7dc458acc64d5cc7e389178ab4bfc2661200c272edac9839a2111bfc9
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: X-Amz-Date: 20180808T155636Z
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Accept-Encoding: gzip
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: -----------------------------------------------------
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: caused by: Post https://sts.us-east-1.amazonaws.com/: dial tcp: lookup sts.us-east-1.amazonaws.com on 10.20.199.19:53: write udp 192.168.234.35:59290->10.20.199.19:53: write: host is down
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 1
2018-08-08T09:56:36.806-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-08T09:56:36.806-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------

(Maintainer edit note: edited to use triple backticks for formatting instead of single backticks)

bflad commented 6 years ago

Hi @sureshoao 👋 Sorry you're running into trouble here. Can you confirm a few things about your setup?

@jjkirby I suspect you're having a separate issue.

The original report seems to point at something closing the connection with EOF (end of file):

...: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
...: caused by: Post https://sts.amazonaws.com/: EOF

While your report seems to point at the DNS server not responding:

...: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
...: caused by: Post https://sts.us-east-1.amazonaws.com/: dial tcp: lookup sts.us-east-1.amazonaws.com on 10.20.199.19:53: write udp 192.168.234.35:59290->10.20.199.19:53: write: host is down

You'll want to triple check via dig @10.20.199.19 sts.us-east-1.amazonaws.com from where Terraform is running in your case. It might have also been a transient issue with your DNS setup. We might be able to reduce the retry threshold similar to how we did for non-existent service endpoints to also include this type of error message for the DNS server not being available so it errors quicker in that case.

jjkirby commented 6 years ago

bflad - Thanks for getting back. dig @10.20.199.19 sts.us-east-1.amazonaws.com resolves perfectly as well as dig @10.20.199.19 sts.amazonaws.com. And yes sts is turned on. I have tried to execute both at my home and work networks. Same result. What did you mean by a transient issue with your DNS setup?

I also found these interesting: https://github.com/golang/dep/issues/1838 and https://github.com/golang/dep/pull/1839. If it is a go DNS issue I would imagine someone on Mac OS X would have a similar problem?

jjkirby commented 6 years ago

@bflad -

I wrote a little go program thinking if it was a go issue:


import (
    "fmt"
    "net"
    "os"
)

func main() {
    ips, err := net.LookupIP("sts.amazonaws.com")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
    fmt.Println(ips)
}

Returned: [54.239.29.25]

So in this case DNS works. Hmmmm

jjkirby commented 6 years ago

Case solved! Little Snitch was blocking because it didn't have a code signature. I turned off network filtering completely even though thought I had a rule to allow. Sorry for false alarm

bflad commented 6 years ago

Closing due to lack of response from original author.

jkodroff commented 5 years ago

Just a note for anyone else experiencing this:

I seem to have this issue when working with Terraform on Xfinity (Comcast) Wi-Fi. (Comcast offers customers the ability to use a public-ish wi-fi network.) Switching to my proper home network seems to have fixed the issue.

Gowiem commented 5 years ago

Another note for folks experiencing this issue -- If you have recently set ENV vars for AWS_SESSION_TOKEN or AWS_ACCESS_KEY_ID and similar, but you're using the credentials file to pull your terraform user creds then you'll hit this when that ENV variable session token expires. Use unset to remove those ENV vars and your terraform config will go back to normal.

ghost commented 5 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!