Closed vmorkunas closed 4 years ago
New findings:
2020/05/04 14:16:18 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/CreateRoute Details:
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Length: 223
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T14:16:18.478Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 14:16:18 GMT
Create route request was sent at 14:16:18 GMT. After that terraform performs this operation:
2020/05/04 14:16:18 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeRouteTables Details:
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Transfer-Encoding: chunked
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T14:16:18.500Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 14:16:17 GMT
AWS API response time is 1 second behind then request before (ec2/CreateRoute): 14:16:17 GMT
Also It's strange that retry call is not executed in this case
There have been other reports of this issue going back to 2016, but, it looks like this very intermittent issue can, as per my colleague's initial report above, still occur. We have seen it happen for a few hrs in a row, e.g., 2 in 10 runs, with repeated apply & destroy cycles, and then no errors for hours or days afterward without any changes to the TF code.
Older references to issue can be found here: https://github.com/hashicorp/terraform/issues/8542
Some of the previous issues, as per the above link (and related links in that issue) stated that the problem was mitigated by adding a create timeout for the route, but, we have > 5m timeout set, and this does not stop the issue from intermittently occurring.
We are now testing with updated TF code that explicitly delays the creation of the routes, until all the other VPC resources are created, e.g., separate TF run to create routes after VPC and related resource creation is completed, e.g., after VPC, NAT, IG and subnets etc are created.
Update:
aws_route resource sometimes crashes while describing route tables with error (due to aws api latency):
Unable to find matching route for Route Table (rtb-08f8b12546e9e1ed8) and destination CIDR block (0.0.0.0/0).
However most of the cases, it's crashing due to latency in aws api, route is not visible in aws yet and it's trying to read route tables already and crashes without finding it on the first try. If retry would be implemented on this case, this could allow to recover after this error and it won't stop terraform run with an error. (now it just gives an error and doesn't retry)
Re: previous comment by my colleague, to be very clear, we are not seeing a "crash", we are seeing the following error:
Unable to find matching route for Route Table (rtb-xxxxxxxxxxxxx) and destination CIDR block (x.x.x.x/x).
This in turn causes our apply to fail
As indicated previously in this thread, we can see a successful response after the route is created, but, the immediate call to describe route tables, produces a response which does not have the newly created route in the RouteSet section of the response, this in turn causes the error referenced above. See request/response snippet below (Route Set should have two entries, not one).
Also noted previously, we’ve have seen cases where the request to describe route tables is out of sync with the route creation, e.g., describe route tables before create route call, though, it usually appears to be in proper order/sequence, but the latency causes an error due to lack of retries of the describe route table call
Again, this issue is very intermittent. Running continuous automated tests over the last 48 hrs, e.g., with repeated, apply/destroy cycles, we have seen no issues for 8-10+ hrs, but then the error will resurface and it can take up to 1.5+ hrs before we can get a successful apply. We've also seen cases, where we recover after a few cycles.
It would appear that a Terraform AWS Provider fix to allow retries for the "describe route tables" call, immediately following the route creation, may resolve this issue.
2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <CreateRouteResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <requestId>2fef32ba-3b32-49af-9316-f0dfd197b8f8</requestId>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <return>true</return>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </CreateRouteResponse>
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] Waiting for state to become: [success]
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeRouteTables Details:
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: POST / HTTP/1.1
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Host: ec2.us-east-1.amazonaws.com
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: User-Agent: aws-sdk-go/1.30.16 (go1.13.7; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.24 (+https://www.terraform.io)
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Length: 82
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAVKX35HH7JKEUER5T/20200504/us-east-1/ec2/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=840a3dc8e6d0a0f9b78f3a59c5bb7e4b3a26a985b659402e1bc62ae82e8bbc39
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: X-Amz-Date: 20200504T233053Z
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Accept-Encoding: gzip
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Action=DescribeRouteTables&RouteTableId.1=rtb-09cf32751abfdd23b&Version=2016-11-15
2020-05-04T23:30:53.655Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: -----------------------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeRouteTables Details:
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: ---[ RESPONSE ]--------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: HTTP/1.1 200 OK
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Connection: close
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Transfer-Encoding: chunked
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Content-Type: text/xml;charset=UTF-8
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Date: Mon, 04 May 2020 23:30:53 GMT
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Server: AmazonEC2
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: Vary: accept-encoding
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: X-Amzn-Requestid: 5e275c1d-e3ef-4216-a21d-88136d7e5634
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4:
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: -----------------------------------------------------
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: 2020/05/04 23:30:53 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <DescribeRouteTablesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <requestId>5e275c1d-e3ef-4216-a21d-88136d7e5634</requestId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <routeTableSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <routeTableId>rtb-086a0b6e30db8ca0c</routeTableId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <vpcId>vpc-02a98a411d6d253f5</vpcId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <ownerId>366674262526</ownerId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <routeSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <destinationCidrBlock>10.120.64.0/20</destinationCidrBlock>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <gatewayId>local</gatewayId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <state>active</state>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <origin>CreateRouteTable</origin>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </routeSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <associationSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <routeTableAssociationId>rtbassoc-0498736d99c2acf80</routeTableAssociationId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <routeTableId>rtb-086a0b6e30db8ca0c</routeTableId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <subnetId>subnet-02c04618a02b834e2</subnetId>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <main>false</main>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <associationState>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <state>associated</state>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </associationState>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </associationSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <propagatingVgwSet/>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <tagSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Application Name</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>tmp</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Private</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>yes</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Product</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>CFT</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Environ</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>NewTemp</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Ticket ID</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>BIZ-2345</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Requester</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>vivek</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Account Owner</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>Sushant</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Approver</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>simran</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Product Owner</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>Rahul Bansal @BankOps</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Creator</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>Eric Bloom</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <key>Name</key>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: <value>NewTemp-Data-Private-Routing-1b</value>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </tagSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </item>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </routeTableSet>
2020-05-04T23:30:53.708Z [DEBUG] plugin.terraform-provider-aws_v2.60.0_x4: </DescribeRouteTablesResponse>
2020/05/04 23:30:53 [DEBUG] module.data_vpc.aws_route.gateway_route["us-east-1b"]: apply errored, but we're indicating that via the Error pointer rather than returning it: Unable to find matching route for Route Table (rtb-086a0b6e30db8ca0c) and destination CIDR block (0.0.0.0/0).
2020/05/04 23:30:53 [TRACE] module.data_vpc: eval: *terraform.EvalMaybeTainted
We created a fork, and resolved the issue. The fix was affected by adding a retry of resourceAwsRouteFindRoute -after- the route creation to confirm the route is in the route table.
We've been running automated testing that repeatedly applies and destroys our infrastructure, and have seen no further errors since implementing this fix.
Here is the code snippet from resource_aws_route.go.
err = resource.Retry(d.Timeout(schema.TimeoutCreate), func() *resource.RetryError {
try := 1
maxRetries := 30
for try < maxRetries {
log.Printf("Route table lookup for matching route with CIDR: %s, Route Table Id: %s, try#: %d", v.(string), d.Get("route_table_id").(string), try)
route, err = resourceAwsRouteFindRoute(conn, d.Get("route_table_id").(string), v.(string), "")
if ( route != nil && err == nil ) {
break
}
time.Sleep(10 * time.Second)
try += 1
}
if ( route != nil && err == nil ) {
return nil
}
return resource.RetryableError(err)
})
Has this fix been submitted as a PR yet?
No.
We've been increasingly running into this issue. We also have a CI that repeatedly applies/destroys infrastructure, and it's been failing about 1 out of 3 times due to this exact error.
Worst part is that simply retrying to deploy doesn't work as Terraform tries to create the route again on the second go, but fails as it already exists.
First go:
Error: Unable to find matching route for Route Table (rtb-xxxxx) and destination CIDR block (0.0.0.0/0).
On retry:
Error: Error creating route: RouteAlreadyExists: The route identified by 0.0.0.0/0 already exists.
We posted a solution above. As per previous comment, this required forking the code and building a custom plugin. As also noted, above, we have not yet submitted a PR.
After looking at the code in question, I saw what the issue is. When resourceAwsRouteFindRoute()
returns a nil value (ie: route not found), it was being wrapped as a resource.RetryableError(nil)
value. The lack of an non-nil err
value inside RetryableError
was causing the Retry logic to treat at is if it were an immediate timeout, and not executing any retries.
I submitted #13747 , which puts an actual error inside RetryableError
, thus triggering the expected retries.
The fix for this has been merged and will release with version 2.67.0 of the Terraform AWS Provider, later this week. Thanks to @alewando for the implementation. 👍
This has been released in version 2.67.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.
For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!
Community Note
Terraform Version
Terraform v0.12.24
Affected Resource(s)
Terraform Configuration Files
Debug Output
Panic Output
Expected Behavior
Route was found and added to route table. Retry for Attempting to find route in route table is not happening like it should.
Actual Behavior
Route was not found:
Unable to find matching route for Route Table (rtb-0c6b2875dd3a2331b) and destination CIDR block (0.0.0.0/0).
Steps to Reproduce
Issue is not reproducible easily, it comes sometimes only. I think there are some issues with AWS API. Increasing timeout for aws_route does not solve the problem. It goes away by itself after few hours.
terraform apply
Important Factoids
Retry for Attempting to find route in route table is not happening like it should.
References