nitrocode commented 1 year ago

Confirmation

[X] My issue isn't already found on the issue tracker.
[X] I have replicated my issue using the latest version of the provider and it is still present.

Terraform and Cloudflare provider version

Terraform 1.5.5 Cloudflare provider 4.12.0 latest

Affected resource(s)

cloudflare_ruleset

Terraform configuration files

resource "cloudflare_ruleset" "custom" {
  kind    = "zone"
  name    = "default"
  phase   = "http_request_firewall_custom"
  zone_id = local.zone_id

  # ... 30 rules ...

  # e.g. pre-existing rule
  rules {
    enabled     = true
    action      = "log"
    description = "<snip>"
    expression  = <<-EOT
              <snip>
            EOT
  }

  # added additional rule which caused a drift. could be any rule
  rules {
    enabled     = true
    action      = "log"
    description = "<snip>"
    expression  = <<-EOT
              any(http.request.headers["custom-header"][*] matches ".?")
            EOT
  }

Link to debug output

none

Panic output

N/A

Expected output

No perma drift

Actual output

Perma drift

Steps to reproduce

Add a new rule or modify an existing one
terraform plan

Additional factoids

This happened after we bumped the provider from 3.x to 4.x

We deleted the cloudflare_ruleset and reimported.

If we do not make a change, we see No changes as expected. Any change we make, and we see a bunch of drift between rules, all affecting arguments that we do not set such as id and version

We have 30 something rules and we see these computed arguments changing for every rule.

in 4.12.0 cloudflare provider

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # cloudflare_ruleset.custom will be updated in-place
  ~ resource "cloudflare_ruleset" "custom" {
        id      = "<snip>"
        name    = "default"
        # (3 unchanged attributes hidden)

      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          ~ last_updated = "2023-08-18 16:36:15.176813 +0000 UTC" -> (known after apply)
          ~ ref          = "<snip>" -> (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)
        }
      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          ~ last_updated = "2023-08-18 16:36:15.176813 +0000 UTC" -> (known after apply)
          ~ ref          = "<snip>" -> (known after apply)
          ~ version      = "15" -> (known after apply)
            # (4 unchanged attributes hidden)

          ~ action_parameters {
              + version  = (known after apply)
                # (3 unchanged attributes hidden)
            }

            # (1 unchanged block hidden)
        }
      + rules {
          + action       = "log"
          + description  = "log traffic with headers"
          + enabled      = true
          + expression   = <<-EOT
                any(http.request.headers["custom-header"][*] matches ".?")
            EOT
          + id           = (known after apply)
          + last_updated = (known after apply)
          + ref          = (known after apply)
          + version      = (known after apply)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

In 4.7.1 cloudflare provider

      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          + last_updated = (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)
        }
      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          + last_updated = (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)

          ~ action_parameters {
              + version  = (known after apply)
                # (1 unchanged attribute hidden)
            }

            # (1 unchanged block hidden)
        }
      + rules {
          + action       = "log"
          + description  = "log traffic with headers"
          + enabled      = true
          + expression   = <<-EOT
                any(http.request.headers["custom-header"][*] matches ".?")
            EOT
          + id           = (known after apply)
          + last_updated = (known after apply)
          + ref          = (known after apply)
          + version      = (known after apply)
        }

We also cannot set a lifecycle ignore rule because this does not work for us

  lifecycle {
    ignore_changes = [
      "rules.*.id",
      "rules.*.version",
      "rules.*.last_updated",
      "rules.*.ref",
      # "rules[*].id",
      # "rules.id",
    ]
  }

I'm told this might work but it's not scaleable. We have over 30 rules. I tried this and still see the drift!

  lifecycle {
    ignore_changes = [
      rules[0].id,
      rules[0].version,
      rules[0].last_updated,
      rules[0].ref,
      rules[1].id,
      rules[1].version,
      rules[1].last_updated,
      rules[1].ref,
      rules[2].id,
      rules[2].version,
      rules[2].last_updated,
      rules[2].ref,
      rules[3].id,
      rules[3].version,
      rules[3].last_updated,
      rules[3].ref,
      # ...
      rules[n].id,
      rules[n].version,
      rules[n].last_updated,
      rules[n].ref,
    ]
  }

We did not see this issue in 3.x cloudflare provider

I don't see any DiffSuppressFunc for any of the above drifted computed arguments. Unsure if the drift needs to be suppressed or if it needs to be investigated.

https://github.com/cloudflare/terraform-provider-cloudflare/blob/adbc0accaafe0df39e78fcfd61d092573e13a25d/internal/framework/service/rulesets/resource.go

References

Related issues

Related pr

https://github.com/cloudflare/terraform-provider-cloudflare/pull/2511 released in 4.8.0

From change log

BREAKING CHANGES:

resource/cloudflare_ruleset: Prevent the rule ID, version and last updated attributes from being set (https://github.com/cloudflare/terraform-provider-cloudflare/issues/2511)

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

github-actions[bot] commented 1 year ago

Thank you for reporting this issue! For maintainers to dig into issues it is required that all issues include the entirety of TF_LOG=DEBUG output to be provided. The only parts that should be redacted are your user credentials in the X-Auth-Key, X-Auth-Email and Authorization HTTP headers. Details such as zone or account identifiers are not considered sensitive but can be redacted if you are very cautious. This log file provides additional context from Terraform, the provider and the Cloudflare API that helps in debugging issues. Without it, maintainers are very limited in what they can do and may hamper diagnosis efforts.

This issue has been marked with triage/needs-information and is unlikely to receive maintainer attention until the log file is provided making this a complete bug report.

jessegonzalez-life360 commented 1 year ago

I can provide this, but I'm not comfortable posting that here in a public forum. I'm an enterprise customer, can I create a case?

Rules, IPs, etc. would take a long while to redact.

Log uploaded to case id: #2901904

jessegonzalez-life360 commented 1 year ago

Thanks for the lifecycle hint. It can help with perma drift until a proper solution is identified.

github-actions[bot] commented 1 year ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

nitrocode commented 1 year ago

Marking this issue as unstale

nitrocode commented 1 year ago

cc: @jacobbednarz

DDuarte commented 11 months ago

Confirming this bug as well, any time there's a change to 1 rule, all the other rules in the ruleset get their mentioned fields updated in diff

nitrocode commented 11 months ago

Thanks for confirming @DDuarte.

If you get a chance @jacobbednarz please take a look at this.

It's hard enough that all of our rules have to be in a single resource and it's even harder to parse through the noisy plan to see the real differences without having to pipe our plans through some post-processing.

jessegonzalez-life360 commented 11 months ago

@DDuarte and @nitrocode if you define a unique ref for the rules block, you should no longer see permanent drift.

Edit: Spoke too soon, the issue persists.

gabrielqs commented 10 months ago

+1 dealing with this issue. given that we're migrating from multiple deprecated cloudflare_firewall_rules into a single new cloudflare_ruleset, this makes the plan outputs become really hard to read. currently using a list of 300 attributes being ignored by lifecycle rules 😅

vojkny commented 10 months ago

I tried to fix this something like:

    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }

However there is nothing like [*] in Terraform syntax, so this couldn't work.

nitrocode commented 10 months ago

@gabrielqs could you share what you use? I tried setting the ignore_changes and I could not get it to work.

github-actions[bot] commented 9 months ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

gabrielqs commented 9 months ago

This is still a problem

gabrielqs commented 9 months ago

@nitrocode sorry, only seeing your message now. here's what we had to do to get around this:

  lifecycle {
    ignore_changes = [
      rules[0].id,
      rules[0].version,
      rules[0].last_updated,
      rules[0].ref,
      rules[0].action_parameters[0].version,
      rules[1].id,
      rules[1].version,
      rules[1].last_updated,
      rules[1].ref,
      rules[1].action_parameters[0].version,
      rules[2].id,
      rules[2].version,
      rules[2].last_updated,
      rules[2].ref,
      rules[2].action_parameters[0].version,
      ...

It is terrible, and we need to remind ourselves to add this every time we add a new rule, but at least the drift doesn't show up on every plan

nitrocode commented 9 months ago

I'm happy it works for you but that doesn't work for us. We still see the drift even with explicitly specifying each individual attributes per rule in ignore_changes.

Since these rules are all in a single terraform resource and we cannot get the plan in a reviewable state, we may have to consider moving these rules out of terraform and having them managed with a script that supplants all the rules upon pr merge.

cc @jacobbednarz friendly ping to get cloudflare's feedback here. This is the 2nd highest voted issue and ideally we should be able to manage and review changes within terraform without so much noise.

troymjones commented 8 months ago

I am providing a full TF_LOG=DEBUG for 2 scenarios, both of which are problematic when working with the Cloudflare ruleset resource. Please note that while there is a workaround (manually verifying them), this is a major issue for my team with a hundred rules in a zone and could result in a bad rule change making its way into the WAF.

Scenarios:

A rule is added in the middle of existing rules. Note that it looks like rules are drastically changing and a 'new' rule is being added to the bottom of the list. However, that 'new' rule is just an existing rule at the end. The rules are correctly being shown, but it is extremely confusing. When there are more than a couple rules, it becomes almost impossible to parse through the changes to make sure no mistakes were made. Also, all rules show changes to id, ref, version, etc, further complicating the issue. Link to gist
A rule is modified (ie name, expression, etc) but no other changes are made. This results in the plan showing all rules having changed. Again, if there are more than a couple rules, it makes it almost impossible to verify that the changes desired are what is shown in the plan. Link to gist

nitrocode commented 8 months ago

Please respond @jacobbednarz . This is the second highest voted issue now.

https://github.com/cloudflare/terraform-provider-cloudflare/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc

saintsinnerr commented 8 months ago

@jacobbednarz , Hello,is this bug expected to be fixed?

pranit-p commented 7 months ago

When will this feature be available?

eephillip commented 7 months ago

I tried to fix this something like:
    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }
However there is nothing like [*] in Terraform syntax, so this couldn't work.

pranit-p commented 7 months ago

Yes @eephillip

But, this code is not workable

I tried to fix this something like:
    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }
However there is nothing like [*] in Terraform syntax, so this couldn't work.
Related hashicorp/terraform#24188

github-actions[bot] commented 6 months ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

fcoelho commented 6 months ago

this is still an issue

gabrielqs commented 6 months ago

Still an issue

Anticmos commented 6 months ago

Still an Issue

Anticmos commented 6 months ago

@nitrocode can you provide the TF_LOG=DEBUG output so that they can remove needs-information tag by chance?

nitrocode commented 6 months ago

@Anticmos please see this comment where the debug log was attached to an internal cloudflare case #2901904 (see https://github.com/cloudflare/terraform-provider-cloudflare/issues/2690#issuecomment-1684856165 in Aug 2023)

nitrocode commented 6 months ago

@Anticmos feel free to add another one. 😄

jogarao96 commented 5 months ago

work around is that deleting the rule from cloudflare cli and recreating it through code. altough you might deleted at cloudflare gui we still have that rule exsits in cli version of cloudflare. Fix is that

curl https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/rulesets --header "X-Auth-Key:{you key here}" --header "X-Auth-Email:{your email here}" >rulesets.yml
Identify the thr rule with no discription but the phase you are trying at in my case http_request_firewall_custom
Delete that rule with cloudflare cli using curl --request DELETE https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/rulesets/<RULESETID> --header "X-Auth-Key:{API KEY}" --header "X-Auth-Email:{EMAIL}" you can get rule id from above file
Re run terraform script to create a new rule which doesnt colide with any rule set ID

boudekerk commented 5 months ago

We're also running into this, as others have mentioned this kind of defeats the purpose of using terraform as the plan becomes unreadable. I.e. at least in the UI it's clear which rule is being edited before saving.

boudekerk commented 5 months ago

@gabrielqs We tried your workaround, but we got errors during apply when trying to lifecycle ignore versions and last_updated. Is there anything special you did to get it to work?

I think one issue is that the action parameter version is always being added, but never ends up in the state, so is added again on each subsequent run.

gabrielqs commented 5 months ago

@boudekerk nothing spectial. the only caveat might be we're running a relatively old version of the cloudflare provider, so maybe there's a discrepancy between what that version is doing and yours?

nitrocode commented 5 months ago

@gabrielqs what version are you running? Maybe we can pin until this issue is resolved.

Is it between version 4 and 4.8.0?

github-actions[bot] commented 4 months ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

nitrocode commented 4 months ago

Unstale

github-actions[bot] commented 3 months ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

IonutLupsan commented 3 months ago

Unstale

emilianoangieri commented 3 months ago

anyone got a resolution or a workaround for this?

ariretiarno commented 2 months ago

still an issue

YogitaDSurve commented 1 month ago

Has anyone came up with any workaround for this till it gets fixed by terraform? It's been a year since this issue is raised 😕.

nitrocode commented 1 month ago

Has anyone tried using a separate terraform directory for these rules using the older 3.x provider ?

https://registry.terraform.io/providers/cloudflare/cloudflare/3.35.0/docs/resources/ruleset

This way you can use the 3.x only for the older ruleset resource and use the latest provider for all the other resources

github-actions[bot] commented 1 week ago

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

troymjones commented 4 days ago

This is still an issue

ebachle commented 2 days ago

I'm willing to bet that the new major version of the provider fixes this based on some of what I read here. https://blog.cloudflare.com/automatically-generating-cloudflares-terraform-provider/

Haven't tested it but I likely will in the next few days.

cloudflare / terraform-provider-cloudflare

Drift in `cloudflare_ruleset` for `http_request_firewall_custom` phase for computed args `id`, `last_updated`, `ref`, `version` #2690

Confirmation

Terraform and Cloudflare provider version

Affected resource(s)

Terraform configuration files

Link to debug output

Panic output

Expected output

Actual output

Steps to reproduce

Additional factoids

References

Community Note