[General Usage]: How to stop replacing schema while migrating

[cyclonstep]

Terraform CLI Version

1.9.5

Terraform Provider Version

0.94.1

Terraform Configuration

resource "snowflake_schema" "test" {
  database                    = snowflake_database.test_db.name
  name                        = "TEST"
  data_retention_time_in_days = 1
  depends_on = [
    snowflake_database.test_db
  ]

  lifecycle {
    prevent_destroy = true
  }
}

Category

category:resource

Object type(s)

data_source:schemas

Expected Behavior

The old existing schema will not be replaced by the newest code applications.

Actual Behavior

It will replaced and all of the data missing.

Steps to Reproduce

1. Upgrade the provider versions
2. terraform plan / apply
3. we will see that the schema will be replaced.

How much impact is this issue causing?

High

Logs

No response

Additional Information

Hello, Thank you very much for all hard works in regards to maintaining this terraform providers.

I've been trying to migrate the terraform (especially grant roles) in these past few days from the old 0.71.0 to the latest 0.94.1.

But when I tried to apply the code, looks like we need to replace all of the schemas. I read in migration_guide that there was some breaking changes and refactoring in the schema resources itself. but is there anything that we can do to prevent the replacement of the object itself while applying? Maybe I'm missing something but I cannot find any way to actually mitigate this situations.

What we've been tested:

• follow resource migration guide -> Need replacement
• using terraform import -> Need replacement
• prevent destroy on lifecycle -> Error, still need to be replaced.

I've been testing out the behaviour and if we apply this, it will destroy all of the data in the schema. We understand that we have Undrop and Clone in snowflake to mitigate this situations but in our current state we cannot afford any downtime and complicated side effects because our snowflake environment is being used by myriad of processes in our business.

As always, thank you very much.

[sfc-gh-jcieslak]

Hey @cyclonstep 👋 I tried to migrate the following configuration from v0.71.0 to v0.94.1

resource "snowflake_database" "test_db" {
  name = "test_db_schema_upgrade"
}

resource "snowflake_schema" "test" {
  database                    = snowflake_database.test_db.name
  name                        = "TEST"
}

and got this plan (a little trimmed, because it's big):

Terraform will perform the following actions:

  # snowflake_database.test_db will be updated in-place
  ~ resource "snowflake_database" "test_db" {
      ~ data_retention_time_in_days                   = 1 -> (known after apply)
        id                                            = "test_db_schema_upgrade"
        name                                          = "test_db_schema_upgrade"
        # (17 unchanged attributes hidden)
    }

  # snowflake_schema.test must be replaced
-/+ resource "snowflake_schema" "test" {
      + catalog                                       = (known after apply)
      ~ data_retention_time_in_days                   = 1 -> (known after apply)
      + default_ddl_collation                         = (known after apply)
      ~ describe_output                               = [] -> (known after apply)
      ~ enable_console_output                         = false -> (known after apply)
      + external_volume                               = (known after apply)
      ~ id                                            = "test_db_schema_upgrade|TEST" -> (known after apply)
      ~ is_transient                                  = "false" -> "default" # forces replacement
      ~ log_level                                     = "OFF" -> (known after apply)
      ~ max_data_extension_time_in_days               = 14 -> (known after apply)
        name                                          = "TEST"
      ~ parameters                                    = []
}

It seems like it_transient is causing the schema to be replaced due to our new way of handling Snowflake defaults. I performed the same upgrade, but this time the is_transient field is set explicitly:

resource "snowflake_database" "test_db" {
  name = "test_db_schema_upgrade"
}

resource "snowflake_schema" "test" {
  database                    = snowflake_database.test_db.name
  name                        = "TEST"
  is_transient = false
}

Now, after the upgrade the plan looked like this:

Terraform will perform the following actions:

  # snowflake_database.test_db will be updated in-place
  ~ resource "snowflake_database" "test_db" {
      ~ data_retention_time_in_days                   = 1 -> (known after apply)
        id                                            = "test_db_schema_upgrade"
        name                                          = "test_db_schema_upgrade"
        # (17 unchanged attributes hidden)
    }

  # snowflake_schema.test will be updated in-place
  ~ resource "snowflake_schema" "test" {
      ~ data_retention_time_in_days                   = 1 -> (known after apply)
        id                                            = "test_db_schema_upgrade|TEST"
        name                                          = "TEST"
      ~ show_output                                   = [
          - {
              - created_on      = "2024-08-26 02:03:47.515 -0700 PDT"
              - database_name   = "test_db_schema_upgrade"
              - dropped_on      = "0001-01-01 00:00:00 +0000 UTC"
              - is_current      = false
              - is_default      = false
              - name            = "TEST"
              - owner           = "ACCOUNTADMIN"
              - owner_role_type = "ROLE"
              - retention_time  = "1"
                # (2 unchanged attributes hidden)
            },
        ] -> (known after apply)
      ~ with_managed_access                           = "false" -> "default"
        # (21 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

So what I propose is to specify the is_transient = false in the older version and then upgrade to the new one. That's also a mistake on our end, because we should provide the state upgrader that would automatically change "false" to "default" if nothing is specified in the configuration. We'll try to add that in the v0.95.0, so no manual work will be necessary on your end, but also that version may introduce breaking changes in other resources that you will need to adapt before upgrading. Will either of these solutions work for you?

[cyclonstep]

@sfc-gh-jcieslak Thank you for your answer! Indeed when I was adding is_transient=false to the code itself, the plan now correctly stops replacing the schema itself.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # snowflake_schema.test will be updated in-place
  ~ resource "snowflake_schema" "test" {
        id                                            = "TEST_DB|TEST"
        name                                          = "TEST"
      ~ show_output                                   = [
          - {
              - created_on      = "2022-05-23 15:58:32.497 +0900 JST"
              - database_name   = "TEST_DB"
              - dropped_on      = "0001-01-01 00:00:00 +0000 UTC"
              - is_current      = false
              - is_default      = false
              - name            = "TEST"
              - owner           = "SYSADMIN"
              - owner_role_type = "ROLE"
              - retention_time  = "1"
                # (2 unchanged attributes hidden)
            },
        ] -> (known after apply)
      ~ with_managed_access                           = "false" -> "default"
        # (22 unchanged attributes hidden)
    }

Let me check this further whether or not is there any other side-effect going on. That would be helpful if you can implement the state upgrader in the 0.95.0. Of course I will check what are the breaking changes that will occurred on my end but as for now, I will go with your recommendations for the current version.

[sfc-gh-jcieslak]

Also, keep in mind that before v1.0, we're introducing many breaking changes, so it would be better to upgrade version by version with the migration guide in mind. Making such a big version jump may break many other resources without proper adjustments.

[sfc-gh-jcieslak]

Hey, actually, I went through this problem with the team. Because of the limited Terraform SDK V2 capabilities, we won't be able to guess and fill in default during the upgrade. Currently, filling the value explicitly will be the only way to upgrade the schema with is_transient unset in previous versions.

[cyclonstep]

@sfc-gh-jcieslak Hello Jan, sorry for replying so late. I see, so there is a limitation on the SDK side right. I saw your documentation commits and thank you for the assistance 👍 . I think this would be quite an issue and it is so easy with one careless terraform apply just to replace everything. If there are things that I could help, please let me know.

To anyone who will read this, please read the plan before applying .

[sfc-gh-jcieslak]

Yeah, in the future, we plan to migrate to the newest Terraform Framework SDK, which should allow for more flexibility, but for now, we are very limited and have to make a lot of workarounds. Thank you for your understanding 🙏. If you have any questions, feel free to ask. Otherwise, I guess we can close this thread (It was closed previously accidentally by GitHub bot).