[Bug]: aws_vpc_security_group_ingress_rule does not apply on subsequent 'terraform apply' call

[ns-vpanfilov]

Terraform Core Version

v1.5.0

AWS Provider Version

5.6.2

Affected Resource(s)

• aws_vpc_security_group_ingress_rule
• aws_vpc_security_group_egress_rule

Expected Behavior

One of the following options:

Option 1: Security group should not lose rules that were added by aws_vpc_security_group_ingress_rule after subsequent terraform apply are executed.

Option 2: Per comment below, this is caused by attempting to use the aws_vpc_security_group_ingress_rule and aws_vpc_security_group_egress_rule resources in conjunction with a aws_security_group resource with rules defined in-line. Terraform should throw an error when an aws_security_group resource with rules defined in-line is used in conjunction with aws_vpc_security_group_ingress_rule and/or aws_vpc_security_group_egress_rule

Actual Behavior

• On the first terraform apply, all rules are created
• On the second terraform apply, rules added via aws_vpc_security_group_ingress_rule are removed from security group
• On the third terraform apply, rules are added back
• On the forth, they are removed again ....

Relevant Error/Panic Output Snippet

resource "aws_security_group" "test_sg" {

  vpc_id = aws_vpc.vpc.id
  name   = "test_sg"

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

locals {
  allowed_ips = ["1.1.1.1/32", "2.2.2.2/32"]
}

resource "aws_vpc_security_group_ingress_rule" "test_rule" {

  for_each = toset(local.allowed_ips)

  security_group_id = aws_security_group.test_sg.id

  from_port   = 443
  to_port     = 443
  ip_protocol = "tcp"
  cidr_ipv4   = each.value
}

Terraform Configuration Files

terraform {
  required_version = ">=1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.6.2"
    }
  }
}

provider "aws" {
  region     = "us-east-1"
}

resource "aws_vpc" "vpc" {
  cidr_block           = "10.0.0.0/24"
  enable_dns_support   = true
  enable_dns_hostnames = true
}

resource "aws_security_group" "test_sg" {
  vpc_id = aws_vpc.vpc.id
  name   = "test_sg"

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

locals {
  allowed_ips = ["1.1.1.1/32", "2.2.2.2/32"]
}

resource "aws_vpc_security_group_ingress_rule" "test_rule" {

  for_each = toset(local.allowed_ips)

  security_group_id = aws_security_group.test_sg.id

  from_port   = 443
  to_port     = 443
  ip_protocol = "tcp"
  cidr_ipv4   = each.value
}

Steps to Reproduce

• Run terraform apply; observe TCP 443 rules added to security group
• Run terraform apply again; notice that 443 rules are removed from security group

aws_vpc.vpc: Refreshing state... [id=vpc-066dc056097118d2a]
aws_security_group.test_sg: Refreshing state... [id=sg-0fe04bad22297a0f4]
aws_vpc_security_group_ingress_rule.test_rule["1.1.1.1/32"]: Refreshing state... [id=sgr-0700f3cd8e487722d]
aws_vpc_security_group_ingress_rule.test_rule["2.2.2.2/32"]: Refreshing state... [id=sgr-027585b51238a4310]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_security_group.test_sg will be updated in-place
  ~ resource "aws_security_group" "test_sg" {
        id                     = "sg-0fe04bad22297a0f4"
      ~ ingress                = [
          - {
              - cidr_blocks      = [
                  - "0.0.0.0/0",
                ]
              - description      = ""
              - from_port        = 22
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 22
            },
          - {
              - cidr_blocks      = [
                  - "1.1.1.1/32",
                  - "2.2.2.2/32",
                ]
              - description      = ""
              - from_port        = 443
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 443
            },
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
        ]
        name                   = "test_sg"
        tags                   = {}
        # (7 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
aws_security_group.test_sg: Modifying... [id=sg-0fe04bad22297a0f4]
aws_security_group.test_sg: Modifications complete after 1s [id=sg-0fe04bad22297a0f4]

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @ns-vpanfilov 👋 Thank you for taking the time to raise this! This is caused by attempting to use the aws_vpc_security_group_ingress_rule and aws_vpc_security_group_egress_rule resources in conjunction with a aws_security_group resource with rules defined in-line. In this configuration, the aws_security_group expects that it is entirely responsible for managing the security group's rules, causing a drift in configuration when the aws_vpc_security_group_ingress_rule and aws_vpc_security_group_egress_rule resources add additional rules. This is called out in each of these resources' documentation with the following note (edited for brevity):

You should not use the aws_vpc_security_group_egress_rule and aws_vpc_security_group_ingress_rule resources in conjunction with an aws_security_group resource with in-line rules or with aws_security_group_rule resources defined for the same Security Group, as rule conflicts may occur and rules will be overwritten.

Since the aws_vpc_security_group_egress_rule and aws_vpc_security_group_ingress_rule resources have been added, it is now recommended to use these distinct resources to manage a Security Group's rules, rather than defining them in-line in the aws_security_group resource.

[ns-vpanfilov]

@justinretzolk, thanks for the feedback.

Should aws provider be updated to throw an error when users use incompatible settings?

[mxk]

There does not appear to be any way to detect and correct drift (e.g. manually added rules) when using the recommended aws_vpc_security_group_ingress_rule and aws_vpc_security_group_egress_rule resources (see also #32743). Unless I'm missing something, if you want to ensure that an SG contains only the rules defined in Terraform, then you have to use ingress and egress inline blocks (which seem analogous to managed_policy_arns and inline_policy for aws_iam_role), so I'm confused about the recommendation not to use them.