Unable to find matching route for Route Table (rtb-xxxxx) and destination CIDR block (x.x.x.x/x)

vmorkunas commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.24

provider.aws v2.60.0

Affected Resource(s)

aws_route

Terraform Configuration Files


Module call:

module "routing_ops_data" {
    source = "../../modules/Stack/Routing"
    routing = {
        src_rt_ids = var.stackCommon.ops_private_route_table_ids
        dst_rt_ids = module.data_vpc.private_route_table_ids
        src_cidr = var.stackCommon.ops_vpc_cidr
        dst_cidr = module.data_vpc.vpc.cidr_block
        peering_connection_id = module.peering_ops_data.peering_connection_ids
        name = "ops-data"
    }
    providers = {
        aws.src = aws.ops
        aws.dst = aws.stack
    }
}

Routing module:
resource "aws_route" "destination" {
    timeouts {
        create = "5m"
        delete = "5m"
    }
    provider = aws.dst
    for_each = {for object in var.routing.dst_rt_ids: object.name => object.id}

    route_table_id = each.value
    destination_cidr_block = var.routing.src_cidr
    vpc_peering_connection_id = var.routing.peering_connection_id

}

Debug Output

Panic Output

Expected Behavior

Route was found and added to route table. Retry for Attempting to find route in route table is not happening like it should.

Actual Behavior

Route was not found:

Unable to find matching route for Route Table (rtb-0c6b2875dd3a2331b) and destination CIDR block (0.0.0.0/0).

Steps to Reproduce

Issue is not reproducible easily, it comes sometimes only. I think there are some issues with AWS API. Increasing timeout for aws_route does not solve the problem. It goes away by itself after few hours.

terraform apply

Important Factoids

Retry for Attempting to find route in route table is not happening like it should.

References

vmorkunas commented 4 years ago

New findings:

2020/05/04 14:16:18 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/CreateRoute Details:
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Length: 223
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 14:16:18 GMT

Create route request was sent at 14:16:18 GMT. After that terraform performs this operation:

2020/05/04 14:16:18 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeRouteTables Details:
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Transfer-Encoding: chunked
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 14:16:17 GMT

AWS API response time is 1 second behind then request before (ec2/CreateRoute): 14:16:17 GMT

Also It's strange that retry call is not executed in this case

wolfulve commented 4 years ago

There have been other reports of this issue going back to 2016, but, it looks like this very intermittent issue can, as per my colleague's initial report above, still occur. We have seen it happen for a few hrs in a row, e.g., 2 in 10 runs, with repeated apply & destroy cycles, and then no errors for hours or days afterward without any changes to the TF code.

Older references to issue can be found here: https://github.com/hashicorp/terraform/issues/8542

Some of the previous issues, as per the above link (and related links in that issue) stated that the problem was mitigated by adding a create timeout for the route, but, we have > 5m timeout set, and this does not stop the issue from intermittently occurring.

We are now testing with updated TF code that explicitly delays the creation of the routes, until all the other VPC resources are created, e.g., separate TF run to create routes after VPC and related resource creation is completed, e.g., after VPC, NAT, IG and subnets etc are created.

vmorkunas commented 4 years ago

Update:

aws_route resource sometimes crashes while describing route tables with error (due to aws api latency):

Unable to find matching route for Route Table (rtb-08f8b12546e9e1ed8) and destination CIDR block (0.0.0.0/0).

However most of the cases, it's crashing due to latency in aws api, route is not visible in aws yet and it's trying to read route tables already and crashes without finding it on the first try. If retry would be implemented on this case, this could allow to recover after this error and it won't stop terraform run with an error. (now it just gives an error and doesn't retry)

wolfulve commented 4 years ago

Re: previous comment by my colleague, to be very clear, we are not seeing a "crash", we are seeing the following error:

Unable to find matching route for Route Table (rtb-xxxxxxxxxxxxx) and destination CIDR block (x.x.x.x/x).

This in turn causes our apply to fail

As indicated previously in this thread, we can see a successful response after the route is created, but, the immediate call to describe route tables, produces a response which does not have the newly created route in the RouteSet section of the response, this in turn causes the error referenced above. See request/response snippet below (Route Set should have two entries, not one).

Also noted previously, we’ve have seen cases where the request to describe route tables is out of sync with the route creation, e.g., describe route tables before create route call, though, it usually appears to be in proper order/sequence, but the latency causes an error due to lack of retries of the describe route table call

Again, this issue is very intermittent. Running continuous automated tests over the last 48 hrs, e.g., with repeated, apply/destroy cycles, we have seen no issues for 8-10+ hrs, but then the error will resurface and it can take up to 1.5+ hrs before we can get a successful apply. We've also seen cases, where we recover after a few cycles.

It would appear that a Terraform AWS Provider fix to allow retries for the "describe route tables" call, immediately following the route creation, may resolve this issue.

2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <CreateRouteResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:     <requestId>2fef32ba-3b32-49af-9316-f0dfd197b8f8</requestId>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:     <return>true</return>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </CreateRouteResponse>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] Waiting for state to become: [success]
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeRouteTables Details:
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: POST / HTTP/1.1
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Host: ec2.us-east-1.amazonaws.com
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: User-Agent: aws-sdk-go/1.30.16 (go1.13.7; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.24 (+https://www.terraform.io)
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Length: 82
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAVKX35HH7JKEUER5T/20200504/us-east-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=840a3dc8e6d0a0f9b78f3a59c5bb7e4b3a26a985b659402e1bc62ae82e8bbc39
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: X-Amz-Date: 20200504T233053Z
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Accept-Encoding: gzip
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Action=DescribeRouteTables&RouteTableId.1=rtb-09cf32751abfdd23b&Version=2016-11-15
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: -----------------------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeRouteTables Details:
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Transfer-Encoding: chunked
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 23:30:53 GMT
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Server: AmazonEC2
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Vary: accept-encoding
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: X-Amzn-Requestid: 5e275c1d-e3ef-4216-a21d-88136d7e5634
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: -----------------------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <DescribeRouteTablesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:     <requestId>5e275c1d-e3ef-4216-a21d-88136d7e5634</requestId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:     <routeTableSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:         <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <routeTableId>rtb-086a0b6e30db8ca0c</routeTableId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <vpcId>vpc-02a98a411d6d253f5</vpcId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <ownerId>366674262526</ownerId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <routeSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <destinationCidrBlock>10.120.64.0/20</destinationCidrBlock>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <gatewayId>local</gatewayId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <state>active</state>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <origin>CreateRouteTable</origin>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             </routeSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <associationSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <routeTableAssociationId>rtbassoc-0498736d99c2acf80</routeTableAssociationId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <routeTableId>rtb-086a0b6e30db8ca0c</routeTableId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <subnetId>subnet-02c04618a02b834e2</subnetId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <main>false</main>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <associationState>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                         <state>associated</state>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     </associationState>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             </associationSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <propagatingVgwSet/>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             <tagSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Application Name</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>tmp</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Private</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>yes</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Product</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>CFT</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Environ</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>NewTemp</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Ticket ID</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>BIZ-2345</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Requester</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>vivek</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Account Owner</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>Sushant</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Approver</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>simran</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Product Owner</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>Rahul Bansal @BankOps</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Creator</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>Eric Bloom</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <key>Name</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                     <value>NewTemp-Data-Private-Routing-1b</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:                 </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:             </tagSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:         </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:     </routeTableSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </DescribeRouteTablesResponse>
2020/05/04 23:30:53 [DEBUG] module.data_vpc.aws_route.gateway_route["us-east-1b"]: apply errored, but we're indicating that via the Error pointer rather than returning it: Unable to find matching route for Route Table (rtb-086a0b6e30db8ca0c) and destination CIDR block (0.0.0.0/0).
2020/05/04 23:30:53 [TRACE] module.data_vpc: eval: *terraform.EvalMaybeTainted

wolfulve commented 4 years ago

We created a fork, and resolved the issue. The fix was affected by adding a retry of resourceAwsRouteFindRoute -after- the route creation to confirm the route is in the route table.

We've been running automated testing that repeatedly applies and destroys our infrastructure, and have seen no further errors since implementing this fix.

Here is the code snippet from resource_aws_route.go.

err = resource.Retry(d.Timeout(schema.TimeoutCreate), func() *resource.RetryError {
    try := 1
    maxRetries := 30
    for try < maxRetries {
        log.Printf("Route table lookup for matching route with CIDR: %s, Route Table Id: %s, try#: %d", v.(string), d.Get("route_table_id").(string), try)
        route, err = resourceAwsRouteFindRoute(conn, d.Get("route_table_id").(string), v.(string), "")
        if ( route != nil && err == nil ) {
            break
        }
        time.Sleep(10 * time.Second)
        try += 1
    }
    if ( route != nil && err == nil ) {
        return nil
    }
    return resource.RetryableError(err)
})

alewando commented 4 years ago

Has this fix been submitted as a PR yet?

wolfulve commented 4 years ago

No.

jValdron commented 4 years ago

We've been increasingly running into this issue. We also have a CI that repeatedly applies/destroys infrastructure, and it's been failing about 1 out of 3 times due to this exact error.

Worst part is that simply retrying to deploy doesn't work as Terraform tries to create the route again on the second go, but fails as it already exists.

First go:

Error: Unable to find matching route for Route Table (rtb-xxxxx) and destination CIDR block (0.0.0.0/0).

On retry:

Error: Error creating route: RouteAlreadyExists: The route identified by 0.0.0.0/0 already exists.

wolfulve commented 4 years ago

We posted a solution above. As per previous comment, this required forking the code and building a custom plugin. As also noted, above, we have not yet submitted a PR.

alewando commented 4 years ago

After looking at the code in question, I saw what the issue is. When resourceAwsRouteFindRoute() returns a nil value (ie: route not found), it was being wrapped as a resource.RetryableError(nil) value. The lack of an non-nil err value inside RetryableError was causing the Retry logic to treat at is if it were an immediate timeout, and not executing any retries. I submitted #13747 , which puts an actual error inside RetryableError, thus triggering the expected retries.

bflad commented 4 years ago

The fix for this has been merged and will release with version 2.67.0 of the Terraform AWS Provider, later this week. Thanks to @alewando for the implementation. 👍

ghost commented 4 years ago

This has been released in version 2.67.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashicorp / terraform-provider-aws