hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.33k stars 1.73k forks source link

google_compute_security_policy rules are always recreated #16882

Open RuBiCK opened 10 months ago

RuBiCK commented 10 months ago

Community Note

Terraform Version

❯ terraform -v
Terraform v1.5.4
on darwin_arm64
[...]
+ provider registry.terraform.io/hashicorp/google v5.10.0
+ provider registry.terraform.io/hashicorp/google-beta v5.10.0
[...]

The same versions were also tested with amd64

Affected Resource(s)

Using resource through module GoogleCloudPlatform/cloud-armor/google version 2.0.1

Expected Behavior

Terraform plan should be idempotent after terraform apply

Actual Behavior

Terraform always wants to recreate all rules in the security policy even if no changes are made and after applying and checking that all changes have been made.

Example of one rule:

  # module.<mymodule>.module.security_policy[0].google_compute_security_policy.policy will be updated in-place
  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/<project-id>/global/securityPolicies/<security-policy-id>"
        name        = "<security-policy-name>"
        # (5 unchanged attributes hidden)

      - rule {
          - action      = "throttle" -> null
          - description = "Throttle per api rule for <redacted>" -> null
          - preview     = false -> null
          - priority    = 80030 -> null

          - match {
              - expr {
                  - expression = <<-EOT
                        has(request.headers['Host'])
                                && (
                                  request.headers['Host'].contains('<redacted>')
                                ) && 
                                  request.path.matches('^/.*$')
                    EOT -> null
                }
            }
           [...]
      + rule {
          + action      = "throttle"
          + description = "Throttle per api rule for <redacted>"
          + preview     = false
          + priority    = 80030

          + match {
              + expr {
                  + expression = <<-EOT
                        has(request.headers['Host'])
                                && (
                                  request.headers['Host'].contains('<redacted>')
                                ) && 
                                  request.path.matches('^/.*$')
                    EOT
                }
            }

          + rate_limit_options {
              + conform_action = "allow"
              + enforce_on_key = "XFF_IP"
              + exceed_action  = "deny(429)"

              + rate_limit_threshold {
                  + count        = 500
                  + interval_sec = 60
                }
            }
        }

References

https://github.com/hashicorp/terraform-provider-google/issues/9084 The same issue as described and related issues

b/318850546

mt185252 commented 9 months ago

Facing similar issue with preconfigured_waf_config block being present in new rules being added via the GCP console.

Seems like its happening only with newly created rules. I have a security policy which was created via the GCP console & has a total of 22 rules(again created via the GCP console), one of these rules was added relatively recently when compared to remaining 21.

I am working on terraform import to construct the state of this security policy and I see that after import for this one particular rule, terraform always tries to delete the preconfigured_waf_config block & as a result re-creates that rule. All the other 21 are just fine.

If I run terraform apply and then immediately run terraform plan I see that it again tries to delete the preconfigured_waf_config block.

To confirm my suspicion that this is been happening only with relatively new rules, I performed the below on the mentioned order:

  1. Created a new security policy and added 3 rules to it via the GCP console.
  2. Imported the state using terraform import.
  3. Ran terraform plan (this showed that the preconfigured_waf_config block in the existing rule has to be deleted & hence the rule will be deleted and recreated.)
  4. Ran terraform apply(It was successful)
  5. Immediately ran terraform plan again & had same observations as in step 3.
    
    $ terraform plan -out test -var-file ./vars/dev.tfvars
    module.cloud-armor["test"].google_compute_security_policy.policy: Refreshing state... [id=projects/<project_id>/global/securityPolicies/test]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place

Terraform will perform the following actions:

module.cloud-armor["test"].google_compute_security_policy.policy will be updated in-place

~ resource "google_compute_security_policy" "policy" { id = "projects//global/securityPolicies/test" name = "test"

(5 unchanged attributes hidden)

  - adaptive_protection_config {
      - layer_7_ddos_defense_config {
          - enable = false -> null
        }
    }

  - rule {
      - action      = "allow" -> null
      - description = "test1" -> null
      - preview     = false -> null
      - priority    = 10 -> null

      - match {
          - versioned_expr = "SRC_IPS_V1" -> null

          - config {
              - src_ip_ranges = [
                  - "1.1.1.1",
                  - "2.2.2.2",
                ] -> null
            }
        }

      - preconfigured_waf_config {
        }
    }
  - rule {
      - action      = "allow" -> null
      - description = "test2" -> null
      - preview     = false -> null
      - priority    = 20 -> null

      - match {
          - versioned_expr = "SRC_IPS_V1" -> null

          - config {
              - src_ip_ranges = [
                  - "3.3.3.3",
                  - "4.4.4.4",
                ] -> null
            }
        }

      - preconfigured_waf_config {
        }
    }
  - rule {
      - action      = "deny" -> null
      - description = "Default rule, higher priority overrides it" -> null
      - preview     = false -> null
      - priority    = 2147483647 -> null

      - match {
          - versioned_expr = "SRC_IPS_V1" -> null

          - config {
              - src_ip_ranges = [
                  - "*",
                ] -> null
            }
        }

      - preconfigured_waf_config {
        }
    }
  - rule {
      - action      = "deny(502)" -> null
      - description = "test3" -> null
      - preview     = false -> null
      - priority    = 30 -> null

      - match {
          - versioned_expr = "SRC_IPS_V1" -> null

          - config {
              - src_ip_ranges = [
                  - "5.5.5.5",
                  - "6.6.6.6",
                ] -> null
            }
        }

      - preconfigured_waf_config {
        }
    }
  + rule {
      + action      = "allow"
      + description = "test1"
      + preview     = false
      + priority    = 10

      + match {
          + versioned_expr = "SRC_IPS_V1"

          + config {
              + src_ip_ranges = [
                  + "1.1.1.1",
                  + "2.2.2.2",
                ]
            }
        }
    }
  + rule {
      + action      = "allow"
      + description = "test2"
      + preview     = false
      + priority    = 20

      + match {
          + versioned_expr = "SRC_IPS_V1"

          + config {
              + src_ip_ranges = [
                  + "3.3.3.3",
                  + "4.4.4.4",
                ]
            }
        }
    }
  + rule {
      + action      = "deny"
      + description = "Default rule, higher priority overrides it"
      + preview     = (known after apply)
      + priority    = 2147483647

      + match {
          + versioned_expr = "SRC_IPS_V1"

          + config {
              + src_ip_ranges = [
                  + "*",
                ]
            }
        }
    }
  + rule {
      + action      = "deny(502)"
      + description = "test3"
      + preview     = false
      + priority    = 30

      + match {
          + versioned_expr = "SRC_IPS_V1"

          + config {
              + src_ip_ranges = [
                  + "5.5.5.5",
                  + "6.6.6.6",
                ]
            }
        }
    }

    # (1 unchanged block hidden)
}


Tried a couple of more times and repeatedly saw this issue occurring.

When I create the policy & rules completely using terraform then I dont see this happening([Using module version v1.1.0](https://github.com/GoogleCloudPlatform/terraform-google-cloud-armor/tree/v1.0.0)).
scramblydevops commented 9 months ago

+1

aaronclong commented 8 months ago

There might be some slight overlap here https://github.com/hashicorp/terraform-provider-google/issues/17288

mersive-raypitmon commented 7 months ago

To prevent my rules from being recreated each time, I had to specify these values in each rule that contained a match.expr (I just have two rules):

    preview  = false
    preconfigured_waf_config {}

I also have a default rule with a match.versioned_expr that didn't need these set, as terraform doesn't think that rule needs to be replaced.)

Here are my versions:

$ terraform -version
Terraform v1.5.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v5.14.0
+ provider registry.terraform.io/hashicorp/google-beta v5.14.0
+ provider registry.terraform.io/hashicorp/random v3.6.0

(I am using the google-beta provider for my google_compute_security_policy.)

rojomisin commented 7 months ago

This bug makes managing cloud armor rules in terraform unusable, ie dangerous in production

sarnepalli commented 7 months ago

To prevent my rules from being recreated each time, I had to specify these values in each rule that contained a match.expr (I just have two rules):

    preview  = false
    preconfigured_waf_config {}

This didn't make any difference for me, I still see the rules being recreated.

bradam12 commented 7 months ago

I had a single rule that was constantly changing with preconfigured_waf_config. I changed the priority of that rule in the UI manually, then reapplied the TF. No more reoccurring changes after that.

Edit: that didn't last long. Came back within a couple days.

snek-git commented 6 months ago

Facing the same problem.


  # module.security_policy.google_compute_security_policy.policy will be updated in-place
  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/divdot-production/global/securityPolicies/frontend-host-policy"
        name        = "frontend-host-policy"
        # (4 unchanged attributes hidden)

      - rule {
          - action      = "allow" -> null
          - description = "Allow Requests only from firebase frontend" -> null
          - preview     = false -> null
          - priority    = 1000 -> null

          - match {
              - expr {
                  - expression = <<-EOT
                        request.headers['host'].lower().contains('example.com')
                    EOT -> null
                }
            }
        }
      + rule {
          + action      = "allow"
          + description = "Allow Requests only from firebase frontend"
          + preview     = false
          + priority    = 1000

          + match {
              + expr {
                  + expression = <<-EOT
                        request.headers['host'].lower().contains('example.com')
                    EOT
                }
            }
        }

        # (3 unchanged blocks hidden)
    }

These rules were initially created using terraform and rerunning terraform apply recreates them every time.

haimke111 commented 4 months ago

Hi guys, I'm also facing the same issue, Any news regarding this? For me it seems that the cause is in the preconfigured_waf_config for the rules and in layer_7_ddos_defense_config


  ~ resource "google_compute_security_policy" "policy" {
        id          = "projects/agp-exchange-prod-0u/global/securityPolicies/grafana-office"
        name        = "grafana-office"
        # (5 unchanged attributes hidden)

      - adaptive_protection_config {
          - layer_7_ddos_defense_config {
              - enable          = false -> null
                # (1 unchanged attribute hidden)
            }
        }

      - rule {
          - action      = "allow" -> null
          - description = "Allow only office CIDR" -> null
          - preview     = false -> null
          - priority    = 0 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "x.x.x.x/29",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      - rule {
          - action      = "deny(403)" -> null
          - description = "Default rule, higher priority overrides it" -> null
          - preview     = false -> null
          - priority    = 2147483647 -> null

          - match {
              - versioned_expr = "SRC_IPS_V1" -> null

              - config {
                  - src_ip_ranges = [
                      - "*",
                    ] -> null
                }
            }

          - preconfigured_waf_config {
            }
        }
      + rule {
          + action      = "allow"
          + description = "Allow only office CIDR"
          + preview     = false
          + priority    = 0

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "x.x.x.x/29",
                    ]
                }
            }
        }
      + rule {
          + action      = "deny(403)"
          + description = "Default rule, higher priority overrides it"
          + preview     = (known after apply)
          + priority    = 2147483647

          + match {
              + versioned_expr = "SRC_IPS_V1"

              + config {
                  + src_ip_ranges = [
                      + "*",
                    ]
                }
            }
        }

        # (1 unchanged block hidden)
    }
imrannayer commented 4 months ago

This issue is blocking https://github.com/GoogleCloudPlatform/terraform-google-cloud-armor/issues/94

GangGreenTemperTatum commented 3 months ago

also experience the exact same issue βž• 😒

    terraform_version: 1.8.5
..

- Installed hashicorp/google v5.38.0 (signed by HashiCorp)
- Installing hashicorp/google-beta v5.38.0...
- Installed hashicorp/google-beta v5.38.0 (signed by HashiCorp)
aebrahim commented 3 months ago

I have the same issue, I believe this is because the API to fetch the security policy returns the rules in an unsorted order, so I believe this method should fix it

Kaschman commented 2 months ago

Suggested workaround while this is broken:

Add a lifecycle ignore changes block to google_compute_security_policy: lifecycle { ignore_changes = [rule] }

Define your rules using the google_compute_security_policy_rule resource.

Your rules will be defined by a different resource, and you will ignore the policy's desire to delete them

tonyay163 commented 1 month ago

I would use google_compute_security_policy_rule but it seems to be missing redirect_options for some reason

spinningarrow commented 3 weeks ago

FYI we're using google_compute_security_policy_rule but ran into this problem anyway. Terraform detected a 'change' in the google_compute_security_policy and deleted all the rules that were created using google_compute_security_policy_rules and then did not recreate them. In effect, all our traffic resulted in 403s because all the rules were inexplicably removed.