schema validation error starting from version 1.70.0

[AlexVasiukov]

Some _confluentschema deployed resources became corrupted starting from provider version 1.70.0.

Example: main.tf:

terraform {
  required_providers {
    confluent = {
      source  = "confluentinc/confluent"
      version = "1.68.0"
    }
  }
}

provider "confluent" {
  schema_registry_id            = var.schema_registry_id
  schema_registry_rest_endpoint = var.schema_registry_rest_endpoint
  schema_registry_api_key       = var.schema_registry_api_key
  schema_registry_api_secret    = var.schema_registry_api_secret
}

resource "confluent_schema" "avas-sbx-value-1" {
  subject_name       = "avas-sbx-value"
  schema             = file( join("/", [ path.root, "schemas", "avas-sbx-value-v1.schema.json"]) )
  format             = "JSON"
  hard_delete        = var.hard_delete
  recreate_on_update = true
}

avas-sbx-value-v1.schema.json:

{
  "$id": "/schemas/avas-sbx",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Some description",
  "definitions": {
    "Error": {
      "additionalProperties": true,
      "properties": {
        "errorCode": {
          "type": "string"
        },
        "errorMessage": {
          "type": "string"
        }
      },
      "required": [
        "errorCode"
      ],
      "title": "Error",
      "type": "object"
    }
  },
  "oneOf": [
    {
      "additionalProperties": true,
      "properties": {
        "runName": {
          "type": "string"
        },
        "errorCode": {
          "type": "string"
        },
        "message": {
          "oneOf": [
            {
              "type": "null"
            },
            {
              "type": "string"
            }
          ]
        },
        "libraryPoolId": {
          "oneOf": [
            {
              "type": "null"
            },
            {
              "type": "string"
            }
          ]
        }
      },
      "required": [
        "runName",
        "errorCode"
      ]
    },
    {
      "additionalProperties": true,
      "properties": {
        "runName": {
          "type": "string"
        },
        "errors": {
          "items": {
            "$ref": "#/definitions/Error"
          },
          "type": "array"
        },
        "libraryPoolId": {
          "oneOf": [
            {
              "type": "null"
            },
            {
              "type": "string"
            }
          ]
        }
      },
      "required": [
        "runName",
        "errors"
      ]
    }
  ],
  "title": "title-1",
  "type": "object"
}

variables.tf:

variable "hard_delete" {
  type = bool
}
variable "schema_registry_id" {
  type        = string
}
variable "schema_registry_rest_endpoint" {
  type        = string
}
variable "schema_registry_api_key" {
  type        = string
}
variable "schema_registry_api_secret" {
  type        = string
}

terraform init/terraform plan passed successfully. If we will bump provider version from 1.68.0 to 1.70.0 - we will receive the error for terraform plan:

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: error updating Schema "lsrc-abcdef/avas-sbx-value/100123": reimport the current resource instance and set recreate_on_update = false to evolve a schema using the same resource instance.
│ In this case, on an update resource instance will reference the updated (latest) schema by overriding schema_identifier, schema and version attributes and the old schema will be orphaned.
│
│   with confluent_schema.avas-sbx-value-1,
│   on main.tf line 17, in resource "confluent_schema" "avas-sbx-value-1":
│   17: resource "confluent_schema" "avas-sbx-value-1" {
│
╵

state file (partially masked):

{
  "version": 4,
  "terraform_version": "1.7.5",
  "serial": 5,
  "lineage": "111a1111-b2b2-cc3c-444d-5555e55ee555",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "confluent_schema",
      "name": "avas-sbx-value-1",
      "provider": "provider[\"registry.terraform.io/confluentinc/confluent\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "credentials": [],
            "format": "JSON",
            "hard_delete": true,
            "id": "lsrc-abcdef/avas-sbx-value/100123",
            "metadata": null,
            "recreate_on_update": true,
            "rest_endpoint": null,
            "ruleset": null,
            "schema": "{\n  \"$id\": \"/schemas/avas-sbx\",\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n  \"description\": \"Some description\",\n  \"definitions\": {\n    \"Error\": {\n      \"additionalProperties\": true,\n      \"properties\": {\n        \"errorCode\": {\n          \"type\": \"string\"\n        },\n        \"errorMessage\": {\n          \"type\": \"string\"\n        }\n      },\n      \"required\": [\n        \"errorCode\"\n      ],\n      \"title\": \"Error\",\n      \"type\": \"object\"\n    }\n  },\n  \"oneOf\": [\n    {\n      \"additionalProperties\": true,\n      \"properties\": {\n        \"runName\": {\n          \"type\": \"string\"\n        },\n        \"errorCode\": {\n          \"type\": \"string\"\n        },\n        \"message\": {\n          \"oneOf\": [\n            {\n              \"type\": \"null\"\n            },\n            {\n              \"type\": \"string\"\n            }\n          ]\n        },\n        \"libraryPoolId\": {\n          \"oneOf\": [\n            {\n              \"type\": \"null\"\n            },\n            {\n              \"type\": \"string\"\n            }\n          ]\n        }\n      },\n      \"required\": [\n        \"runName\",\n        \"errorCode\"\n      ]\n    },\n    {\n      \"additionalProperties\": true,\n      \"properties\": {\n        \"runName\": {\n          \"type\": \"string\"\n        },\n        \"errors\": {\n          \"items\": {\n            \"$ref\": \"#/definitions/Error\"\n          },\n          \"type\": \"array\"\n        },\n        \"libraryPoolId\": {\n          \"oneOf\": [\n            {\n              \"type\": \"null\"\n            },\n            {\n              \"type\": \"string\"\n            }\n          ]\n        }\n      },\n      \"required\": [\n        \"runName\",\n        \"errors\"\n      ]\n    }\n  ],\n  \"title\": \"title-1\",\n  \"type\": \"object\"\n}",
            "schema_identifier": 100123,
            "schema_reference": [],
            "schema_registry_cluster": [],
            "subject_name": "avas-sbx-value",
            "version": 1
          },
          "sensitive_attributes": [],
          "private": "abcDef=="
        }
      ]
    }
  ],
  "check_results": null
}

I also noticed that the error may be resolved by compressing schema file in terraform and state (schema section): avas-sbx-value-v1.schema.json:

{"$id":"/schemas/avas-sbx","$schema":"http://json-schema.org/draft-07/schema#","description":"Some description","definitions":{"Error":{"additionalProperties":true,"properties":{"errorCode":{"type":"string"},"errorMessage":{"type":"string"}},"required":["errorCode"],"title":"Error","type":"object"}},"oneOf":[{"additionalProperties":true,"properties":{"runName":{"type":"string"},"errorCode":{"type":"string"},"message":{"oneOf":[{"type":"null"},{"type":"string"}]},"libraryPoolId":{"oneOf":[{"type":"null"},{"type":"string"}]}},"required":["runName","errorCode"]},{"additionalProperties":true,"properties":{"runName":{"type":"string"},"errors":{"items":{"$ref":"#/definitions/Error"},"type":"array"},"libraryPoolId":{"oneOf":[{"type":"null"},{"type":"string"}]}},"required":["runName","errors"]}],"title":"title-1","type":"object"}

state:

{
  "version": 4,
  "terraform_version": "1.7.5",
  "serial": 5,
  "lineage": "111a1111-b2b2-cc3c-444d-5555e55ee555",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "confluent_schema",
      "name": "avas-sbx-value-1",
      "provider": "provider[\"registry.terraform.io/confluentinc/confluent\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "credentials": [],
            "format": "JSON",
            "hard_delete": true,
            "id": "lsrc-abcdef/avas-sbx-value/100123",
            "metadata": null,
            "recreate_on_update": true,
            "rest_endpoint": null,
            "ruleset": null,
            "schema": "{\"$id\":\"/schemas/avas-sbx\",\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"description\":\"Some description\",\"definitions\":{\"Error\":{\"additionalProperties\":true,\"properties\":{\"errorCode\":{\"type\":\"string\"},\"errorMessage\":{\"type\":\"string\"}},\"required\":[\"errorCode\"],\"title\":\"Error\",\"type\":\"object\"}},\"oneOf\":[{\"additionalProperties\":true,\"properties\":{\"runName\":{\"type\":\"string\"},\"errorCode\":{\"type\":\"string\"},\"message\":{\"oneOf\":[{\"type\":\"null\"},{\"type\":\"string\"}]},\"libraryPoolId\":{\"oneOf\":[{\"type\":\"null\"},{\"type\":\"string\"}]}},\"required\":[\"runName\",\"errorCode\"]},{\"additionalProperties\":true,\"properties\":{\"runName\":{\"type\":\"string\"},\"errors\":{\"items\":{\"$ref\":\"#/definitions/Error\"},\"type\":\"array\"},\"libraryPoolId\":{\"oneOf\":[{\"type\":\"null\"},{\"type\":\"string\"}]}},\"required\":[\"runName\",\"errors\"]}],\"title\":\"title-1\",\"type\":\"object\"}",
            "schema_identifier": 100123,
            "schema_reference": [],
            "schema_registry_cluster": [],
            "subject_name": "avas-sbx-value",
            "version": 1
          },
          "sensitive_attributes": [],
          "private": "abcDef=="
        }
      ]
    }
  ],
  "check_results": null
}

^^ In this case terraform plan runs without errors. The check appears to have failed due to space/line break characters.

[linouk23]

Thanks for reporting this issue @AlexVasiukov! We did fix #318 and #293 in 1.70.0 (context).

Could you try to run terraform refresh -target=confluent_schema.avas-sbx-value-1 (make sure to create a backup for your TF state file before running this command)? Thank you!

[AlexVasiukov]

@linouk23, sure, here is an output:

terraform refresh -target=confluent_schema.avas-sbx-value-1 -var-file=local-variables.tfvars -state=avas-sbx.json

╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.      
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically       
│ suggests to use it as part of an error message.
╵
╷
│ Error: error updating Schema "lsrc-abcdef/avas-sbx-value/100123": reimport the current resource instance and set recreate_on_update = false to evolve a schema using the same resource instance.
│ In this case, on an update resource instance will reference the updated (latest) schema by overriding schema_identifier, schema and version attributes and the old schema will be orphaned.
│
│   with confluent_schema.avas-sbx-value-1,
│   on main.tf line 17, in resource "confluent_schema" "avas-sbx-value-1":
│   17: resource "confluent_schema" "avas-sbx-value-1" {
│
╵

The full debug could be found in attach tf-refresh.txt

[AlexVasiukov]

Hi @linouk23, Do you have any updates regarding this issue?