Terraform State Out of Sync with Panorama

[Eric-Jckson]

Describe the bug

The Terraform Apply execution failed halfway through the run using the resource "panos_panorama_security_rule_group" . This caused some of the security policy rules to be created in panorama but was not reflected in the terraform state.

Initializing plugins and modules...
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Creating...
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [10s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [20s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [30s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [40s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [50s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m0s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m10s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m20s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m30s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m40s elapsed]
module.config-securitypolicy-cep.module.securitypolicy.panos_panorama_security_rule_group.default["0"]: Still creating... [1m50s elapsed]

Error:  Global Citrix Web Req -> destination is invalid

  on modules/terraform-panorama-config-securitypolicy-spw-cep/modules/terraform-panorama-config-securitypolicy-cep/main.tf line 1, in resource "panos_panorama_security_rule_group" "default":
   1: resource "panos_panorama_security_rule_group" "default" {

Terraform State

{
  "version": 4,
  "terraform_version": "0.13.5",
  "serial": 0,
  "lineage": "2816920a-9647-09b8-5cc6-8bce9128dbd4",
  "outputs": {},
  "resources": []
}

Expected behavior

The rules that have been created in panorama should be reflected in the state.

Current behavior

Rules are created in panorama but not managed with the terraform state.

Possible solution

Steps to reproduce

1. Terraform Apply
2. Error Occurs

Context

Your Environment

Terraform Enterprise Panorama

[shinmog]

@Eric-Jckson

Ok, so the security rule resources have been majorly overhauled in v1.9.x with the addition of audit comments and the preservation of names. Even with all those changes, a desync between what's in state and what's on Panorama is going to be possible.

In particular with the policy rule group resources, a desync between what's on live and what's in state is totally possible and is honestly just part of the how those resources work. Imagine that you have a rule group [A, B, C] defined, but then someone manually adds a rule between so now on PAN-OS it looks like [A, B, x, C]. When Terraform refreshes, it will report that your rule group looks like [A, B]. This won't matter to terraform apply, btw, because when the provider goes to configure the group, it will see that C exists (and has the correct configuration in our hypothetical example), so it just needs to be moved after B.

In your case, you are creating the rule group, not updating, but the logic is the same. When you fix the definition of your rule group, after terraform apply the provider will see that there are a bunch of rules that have the correct definition and will not alter them. This in turn will reduce the size/count of the API calls that the provider has to make to configure the remaining rules.

[Eric-Jckson]

Hey @shinmog thanks for the detailed reply! I have been testing the new v1.9.0 panos version with the policy rule group and it is all working as intended. This has been a major improvement since the entire rule group does not have to be deleted and recreated. The only issue that we are encountering now is the limit that terraform restricts for the size of data passing through the provider.

https://github.com/hashicorp/terraform-plugin-go/issues/106

Since we have 3,000 rules we need to split them up in different resource definitions. Would you still suggest that we continue to use the policy rule group to define these rules or should we be using the single policy rule resource?

[shinmog]

You'll be worse off trying to use panos_security_policy / panos_panorama_security_policy, as the 4M limit applies no matter what. Besides HashiCorp expanding the resource size limit from 4M to something higher, you'll have no choice but to break them into multiple panos_security_rule_group / panos_panorama_security_rule_group resource blocks.