data-henrik commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

$ terraform -v
Terraform v1.3.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/kubernetes v2.11.0
+ provider registry.terraform.io/hashicorp/null v3.2.0
+ provider registry.terraform.io/ibm-cloud/ibm v1.47.1

Affected Resource(s)

ibm_cbr_rule

Terraform Configuration Files

This is the resource which I could apply n times, like in an infinite loop.

# registry access
resource "ibm_cbr_rule" "cbr_rule_registry" {
  contexts {
    attributes {
      name  = "networkZoneId"
      value = ibm_cbr_zone.cbr_zone_k8s.id
    }
  }

  description      = "restrict access to registry to cluster"
  enforcement_mode = var.cbr_enforcement_mode
  resources {
    attributes {
      name  = "accountId"
      value = data.ibm_iam_account_settings.team_iam_account_settings.account_id
    }
    attributes {
      name     = "resourceType"
      operator = "stringEquals"
      value    = "namespace"
    }
    attributes {
      name     = "resource"
      operator = "stringEquals"
      value    = "henrik/bluegreen"
    }
    attributes {
      name     = "serviceName"
      operator = "stringEquals"
      value    = "container-registry"
    }

  }
}

Debug Output```

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place

Terraform will perform the following actions:

ibm_cbr_rule.cbr_rule_registry will be updated in-place

~ resource "ibm_cbr_rule" "cbr_rule_registry" { id = "31e9546169c70af51a0899ad8d7d4e8d"

(9 unchanged attributes hidden)

  ~ resources {
      ~ attributes {
          ~ name     = "resource" -> "resourceType"
          ~ value    = "henrik/bluegreen" -> "namespace"
            # (1 unchanged attribute hidden)
        }
      ~ attributes {
          ~ name     = "resourceType" -> "resource"
          ~ value    = "namespace" -> "henrik/bluegreen"
            # (1 unchanged attribute hidden)
        }

        # (2 unchanged blocks hidden)
    }

    # (1 unchanged block hidden)
}

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.

Enter a value: yes

ibm_cbr_rule.cbr_rule_registry: Modifying... [id=31e9546169c70af51a0899ad8d7d4e8d] ibm_cbr_rule.cbr_rule_registry: Modifications complete after 1s [id=31e9546169c70af51a0899ad8d7d4e8d]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.


<!---
Please provide a link to a GitHub Gist containing the complete debug output. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

To obtain the debug output, see the [Terraform documentation on debugging](https://www.terraform.io/docs/internals/debugging.html).
--->

### Panic Output

<!--- If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the `crash.log`. --->

### Expected Behavior
I expect the resource, when applied once, to be stable. When I call "apply" again there should be not changes.
<!--- What should have happened? --->

### Actual Behavior
Whenever I apply again, a change is detected and the resource updated. The ID remains the same. On the next apply, the same resource is updated again.
<!--- What actually happened? --->

### Steps to Reproduce

<!--- Please list the steps required to reproduce the issue. --->

1. `terraform apply`

### Important Factoids

<!--- Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? --->

### References

<!---
Information about referencing Github Issues: https://help.github.com/articles/basic-writing-and-formatting-syntax/#referencing-issues-and-pull-requests

Are there any other GitHub issues (open or closed) or pull requests that should be linked here? Vendor documentation? For example:
--->

* #0000

daniel-butler-irl commented 1 year ago

I am also seeing the same issue and it creates problems for me since I have tests to ensure my modules are idempotent. This breaks me. Here is a full end-to-end example


terraform {
  required_version = ">= 1.0.0"
  required_providers {
    # Pin to the lowest provider version of the range defined in the main module to ensure lowest version still works
    ibm = {
      source  = "ibm-cloud/ibm"
      version = "1.45.0"
    }
  }
}

variable "ibmcloud_api_key" {
  type        = string
  description = "The IBM Cloud API Token"
  sensitive   = true
}

provider "ibm" {
  ibmcloud_api_key = var.ibmcloud_api_key
}

resource "ibm_resource_group" "resourceGroup" {
  name     = "test-rg-cbr"
}

resource "ibm_is_vpc" "example_vpc" {
  name           = "test-vpc-cbr"
  resource_group = ibm_resource_group.resourceGroup.id
}

resource "ibm_is_public_gateway" "testacc_gateway" {
  name           = "test-pgateway-cbr"
  vpc            = ibm_is_vpc.example_vpc.id
  zone           = "us-south-1"
  resource_group = ibm_resource_group.resourceGroup.id
}

resource "ibm_is_subnet" "testacc_subnet" {
  name                     = "test-subnet-cbr"
  vpc                      = ibm_is_vpc.example_vpc.id
  zone                     = "us-south-1"
  public_gateway           = ibm_is_public_gateway.testacc_gateway.id
  total_ipv4_address_count = 256
  resource_group           = ibm_resource_group.resourceGroup.id
}

resource "ibm_database" "redis_database" {
  resource_group_id                    = ibm_resource_group.resourceGroup.id
  name                                 = "test-redis-cbr"
  service                              = "databases-for-redis"
  plan                                 = "standard"
  location                             = "us-south"
}

data "ibm_iam_account_settings" "iam_account_settings" {
}

resource "ibm_cbr_zone" "cbr_zone" {
  account_id = data.ibm_iam_account_settings.iam_account_settings.id
  addresses {
    type  = "vpc" # to bind a specific vpc to the zone
    value = ibm_is_vpc.example_vpc.crn
  }
  description = "this is an example of zone"

  name = "test-redis-zone"
}

resource "ibm_cbr_rule" "cbr_rule" {
  contexts {
    attributes {
      name  = "networkZoneId"
      value = ibm_cbr_zone.cbr_zone.id
    }
  }
  description = "this is an example of rule"
  enforcement_mode = "enabled"
  operations {
    api_types {
      api_type_id = "crn:v1:bluemix:public:context-based-restrictions::::api-type:data-plane"
    }
  }
  resources {
    attributes {
      name     = "accountId"
      value    = data.ibm_iam_account_settings.iam_account_settings.id
      operator = "stringEquals"
    }
    attributes {
      name     = "serviceName"
      value    = "databases-for-redis"
      operator = "stringEquals"
    }
    attributes {
      name     = "serviceInstance"
      value    = ibm_database.redis_database.id
      operator = "stringEquals"
    }

  }
}

Every time after applying if we do a plan changes are detected. The change is the order in which the attributes return is different every time. Even if we apply the change next time a change is detected. Here is the output:

❯ terraform plan
data.ibm_iam_account_settings.iam_account_settings: Reading...                                                                       
ibm_resource_group.resourceGroup: Refreshing state... [id=e9957c89eed5494bb717c73a0ddcca8d]
data.ibm_iam_account_settings.iam_account_settings: Read complete after 0s [id=abac0df06b644a9cabc6e44f55b3880e]
ibm_is_vpc.example_vpc: Refreshing state... [id=r006-577ceb53-30b3-4b01-b981-3fb6ea39d157]
ibm_database.redis_database: Refreshing state... [id=crn:v1:bluemix:public:databases-for-redis:us-south:a/abac0df06b644a9cabc6e44f55b3880e:3990ce80-133e-4d05-8bb8-36c608dbe144::]
ibm_is_public_gateway.testacc_gateway: Refreshing state... [id=r006-26536c02-cb1f-4d42-8b51-07c120713f4b]
ibm_cbr_zone.cbr_zone: Refreshing state... [id=366d56825bd272e7e7ffe0299e10581c]
ibm_cbr_rule.cbr_rule: Refreshing state... [id=4fbe9ab9d9277c48b9a024aa0a1a7dca]
ibm_is_subnet.testacc_subnet: Refreshing state... [id=0717-c99e549c-68c6-477a-ad9b-935f8c8cdd60]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # ibm_cbr_rule.cbr_rule will be updated in-place
  ~ resource "ibm_cbr_rule" "cbr_rule" {
        id                  = "4fbe9ab9d9277c48b9a024aa0a1a7dca"
        # (9 unchanged attributes hidden)

      ~ resources {
          ~ attributes {
              ~ name     = "serviceInstance" -> "serviceName"
              ~ value    = "crn:v1:bluemix:public:databases-for-redis:us-south:a/abac0df06b644a9cabc6e44f55b3880e:3990ce80-133e-4d05-8bb8-36c608dbe144::" -> "databases-for-redis"
                # (1 unchanged attribute hidden)
            }
          ~ attributes {
              ~ name     = "serviceName" -> "serviceInstance"
              ~ value    = "databases-for-redis" -> "crn:v1:bluemix:public:databases-for-redis:us-south:a/abac0df06b644a9cabc6e44f55b3880e:3990ce80-133e-4d05-8bb8-36c608dbe144::"
                # (1 unchanged attribute hidden)
            }

            # (1 unchanged block hidden)
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

ocofaigh commented 1 year ago

It may be the same root cause, but I am seeing the same issue with ibm_schematics_workspace resource. After initial terraform apply has completed, running a subsequent apply, terraform is picking up an update operation, however there is no actual update. The order of the value for template_env_settings is being detected as an update, even though values are the same, but just in a different order:

Before:

{"template_env_settings":[{"__netrc__":"[[\"github.ibm.com\",\"username\",\"git-token\"]]"},{"API_DATA_IS_SENSITIVE":"true"}]}

After:

{"template_env_settings":[{"API_DATA_IS_SENSITIVE":"true"},{"__netrc__":"[[\"github.ibm.com\",\"username\",\"git-token\"]]"}]}`

We have tests to check that a module is idempotent, and this bug is failing our tests.

hkantare commented 1 year ago

@zhenwan Can you please look into the issue

zhenwan commented 1 year ago

@data-henrik The order of attributes block in your resource config is different from the one returned from CBR service. you can find out the order of attributes returned from CBR service in the terraform.tfstate file.

To prevent this behavior from happening, you can change order of the attributes to be the same as terraform.tfstate file:

# registry access
resource "ibm_cbr_rule" "cbr_rule_registry" {
  contexts {
    attributes {
      name  = "networkZoneId"
      value = ibm_cbr_zone.cbr_zone_k8s.id
    }
  }

  description      = "restrict access to registry to cluster"
  enforcement_mode = var.cbr_enforcement_mode
  resources {
    attributes {
      name  = "accountId"
      value = data.ibm_iam_account_settings.team_iam_account_settings.account_id
    }
    attributes {
      name     = "resource"
      operator = "stringEquals"
      value    = "henrik/bluegreen"
    }
    attributes {
      name     = "resourceType"
      operator = "stringEquals"
      value    = "namespace"
    }
    attributes {
      name     = "serviceName"
      operator = "stringEquals"
      value    = "container-registry"
    }

  }
}

data-henrik commented 1 year ago

That might be a workaround to cope with this bug. But it is not a solution. The order should not matter.

hkantare commented 1 year ago

@zhenwan That a workaround may can you look into changing the Type from list to set . As per Terraform development we should Type set in resources instead of list because of the order issue.