[Bug]: some VPC resources leaking during creation

[woranhun]

Terraform Core Version

v1.9.7

AWS Provider Version

5.72.0

Affected Resource(s)

• aws_subnet
• aws_ec2_subnet_cidr_reservation
• aws_security_group_rule
• aws_network_acl_rule

Expected Behavior

If a resource is created on AWS side, then it should not be leaked.

Actual Behavior

This is can happen because conn.CreateSubnet returns earlier during an error and therefore the subnedId is not saved. https://github.com/hashicorp/terraform-provider-aws/blob/12339906a1803312219d9bfd257c025aaad05513/internal/service/ec2/vpc_subnet.go#L197

Relevant Error/Panic Output Snippet

output, err := conn.CreateSubnet(ctx, input)

    if err != nil {
      return sdkdiag.AppendErrorf(diags, "creating EC2 Subnet: %s", err)
    }

    d.SetId(aws.ToString(output.Subnet.SubnetId))

Terraform Configuration Files

    resource "aws_subnet" "main" {
    vpc_id     = "vpc-01ced180f39a4a9d2"
    cidr_block = "10.0.1.0/24"

    tags = {
      Name = "Main"
    }
  }

Steps to Reproduce

• Create the terraform resource

  resource "aws_subnet" "main" {
  vpc_id     = "vpc-01ced180f39a4a9d2"
  cidr_block = "10.0.1.0/24"
  
  tags = {
    Name = "Main"
  }
  }

• Terraform apply the resource and after a few seconds CTRL+C the execution. (This can also happen, when for any reason the conn.CreateSubnet(ctx, input) returns with an error. ) Apply it again and a resource creation fails, because a subnet with the CIDR already exists. The subnet-id was leaked during the first creation.

  ~/projects/core/terraform-provider-aws-test terraform apply                                                                                                                                
  ╷
  │ Warning: Provider development overrides are in effect
  │ 
  │ The following provider development overrides are set in the CLI configuration:
  │  - hashicorp/aws in /home/ec2-user/go/bin
  │ 
  │ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
  ╵
  
  Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  
  Terraform will perform the following actions:
  
  # aws_subnet.main will be created
  + resource "aws_subnet" "main" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = (known after apply)
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name" = "Main"
        }
      + tags_all                                       = {
          + "Name" = "Main"
        }
      + vpc_id                                         = "vpc-01ced180f39a4a9d2"
    }
  
  Plan: 1 to add, 0 to change, 0 to destroy.
  
  Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  
  Enter a value: yes
  
  aws_subnet.main: Creating...
  ^C
  Interrupt received.
  Please wait for Terraform to exit or data loss may occur.
  Gracefully shutting down...
  
  Stopping operation...
  ╷
  │ Error: execution halted
  │ 
  │ 
  ╵
  ╷
  │ Error: execution halted
  │ 
  │ 
  ╵
  ╷
  │ Error: creating EC2 Subnet: operation error EC2: CreateSubnet, https response error StatusCode: 0, RequestID: , canceled, context canceled
  │ 
  │   with aws_subnet.main,
  │   on main.tf line 16, in resource "aws_subnet" "main":
  │   16: resource "aws_subnet" "main" {
  │ 
  ╵
  ~/projects/core/terraform-provider-aws-test terraform apply                                                                                                                                
  ╷
  │ Warning: Provider development overrides are in effect
  │ 
  │ The following provider development overrides are set in the CLI configuration:
  │  - hashicorp/aws in /home/ec2-user/go/bin
  │ 
  │ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
  ╵
  
  Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  
  Terraform will perform the following actions:
  
  # aws_subnet.main will be created
  + resource "aws_subnet" "main" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = (known after apply)
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name" = "Main"
        }
      + tags_all                                       = {
          + "Name" = "Main"
        }
      + vpc_id                                         = "vpc-01ced180f39a4a9d2"
    }
  
  Plan: 1 to add, 0 to change, 0 to destroy.
  
  Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  
  Enter a value: yes
  
  aws_subnet.main: Creating...
  ╷
  │ Error: creating EC2 Subnet: operation error EC2: CreateSubnet, https response error StatusCode: 400, RequestID: 4498f299-ecea-4d03-b188-bae726938f69, api error InvalidSubnet.Conflict: The CIDR '10.0.1.0/24' conflicts with another subnet
  │ 
  │   with aws_subnet.main,
  │   on main.tf line 16, in resource "aws_subnet" "main":
  │   16: resource "aws_subnet" "main" {
  │ 
  ╵

• This is can happen because conn.CreateSubnet returns earlier during an error and therefore the subnedId is not saved. https://github.com/hashicorp/terraform-provider-aws/blob/12339906a1803312219d9bfd257c025aaad05513/internal/service/ec2/vpc_subnet.go#L197

  output, err := conn.CreateSubnet(ctx, input)
  
  if err != nil {
    return sdkdiag.AppendErrorf(diags, "creating EC2 Subnet: %s", err)
  }
  
  d.SetId(aws.ToString(output.Subnet.SubnetId))

Debug Output

No response

Panic Output

No response

Important Factoids

We have seen issues with only the 4 resources I mentioned before. I think because these are the ones we create in high quantity. However, I think other resources might be affected as well...

References

relates to: https://github.com/crossplane-contrib/provider-upjet-aws/issues/1482 relates to: https://github.com/hashicorp/terraform-provider-aws/issues/38251

Would you like to implement a fix?

Yes

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[alexbacchin]

@woranhun seems to me a normal behavior of the EC2 API. https://docs.aws.amazon.com/ec2/latest/devguide/eventual-consistency.html

[woranhun]

@alexbacchin Can you elaborate on why do you think is it a normal behavior? I think, It is the same issue as happened before with RDS (https://github.com/hashicorp/terraform-provider-aws/issues/38251). And also this behavior causes issues with managing AWS resources from Crossplane (https://github.com/crossplane-contrib/provider-upjet-aws/issues/1482)

[alexbacchin]

@woranhun When you CTRL+C Terraform, but the AWS API action is successfully received, AWS control plane will execute the action and the Terraform state will have no record of the resource been successfully created. Thus, it will try again in the subsequent apply, as there is already a subnet with the same CIDR, you get an error.

[woranhun]

@alexbacchin Yes, but If a resource creation was triggered from TF, then TF should be aware that the resource exists, because it was created earlier by itself.

  output, err := conn.CreateSubnet(ctx, input)

  if err != nil {
    return sdkdiag.AppendErrorf(diags, "creating EC2 Subnet: %s", err)
  }

  d.SetId(aws.ToString(output.Subnet.SubnetId))

For example in this case: The resource was created in AWS AND err is not nil (for whatever reason), then the SubnetId is lost forever (because of the return).

My plan is to move d.SetId(aws.ToString(output.Subnet.SubnetId)) line above to the error check.