Ensure resources are validated while running terraform plan

[Matt2298]

Problem

When creating terraform resources for kafka, some of the resources (the only ones I've stumbled across so far) have hard limits on the length of the field. This hard limit is only found when running terraform apply, and does not appear when running terraform plan.

This means that when running terraform in a generic CI pipeline, the pre-merge checks which run terraform plan and succeed, however the subsequent merged pipeline that applies the changes would fail.

Expected

When running terraform plan, resource validation should raise errors as it does in terraform plan, so that they can be caught and rectified before merging.

Steps to reproduce:

• Create a kafka resource such as an ACL with a description that is longer than 128 characters long:

  resource "confluent_service_account" "this" {
  display_name = "my-service-account"
  description  = "abc....(128 CHARS LONG)"
  }

• Run terraform plan Output:

  # module.this.module.my-service-account.confluent_service_account.this will be created
  + resource "confluent_service_account" "this" {
    + api_version  = (known after apply)
    + description  = "abc....(128 CHARS LONG)"
    + display_name = "my-service-account"
    + id           = (known after apply)
    + kind         = (known after apply)
  }

• Run terraform apply Output:

  │ Error: error creating Service Account "my-service-account": 400 Bad Request: description limit to 128 characters.
  │ 
  │   with module.this.module.my-service-account.this,
  │   on ../module/connector_datagen/auth.tf line 4, in resource "confluent_service_account" "this":
  │    4: resource "confluent_service_account" "this" {

Notes

This affects both the description of the resources at 128 chars, as well as the names for connectors at 64 chars:

│ 1. "name" : ["\"name\" should be not longer than 64"]

[nicolasyanncouturier]

Some I've stumbled upon myself (exact same situation as described: CI the pre-merge checks which run terraform plan all OK, then the subsequent merged pipeline that applies the changes would fail)

• confluent_identity_pool:
  • display_name: limited to 64
  • description: limited to 128
  • filter: limited to 300

[Nemeczek]

Just to add to this. Recently we were adding topic ACL and one of them was: "'internal.system". Which is clearly wrong. But our PR pipeline actually does plan the changes: resource_name = "'internal.system" And then of course apply falied: Error: error creating Kafka ACLs: 400 Bad Request: Internal error: Could not create ACLs because following resource names are invalid : ['internal.system]

I think basic validation like topics/ACL length and shape validation should be part of this provider because right now it feels more like weaving jsons not code.