Google Cloud Domains Registration parameter values keep getting out of sync and failing to redeploy

fathom-aniket commented 1 month ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.6.3 on linux_amd64

provider registry.terraform.io/hashicorp/google v5.25.0
provider registry.terraform.io/hashicorp/google-beta v5.25.0

Affected Resource(s)

google_clouddomains_registration

Terraform Configuration

{
  "//": {
    "metadata": {
      "backend": "gcs",
      "stackName": "airflow-setup",
      "version": "0.20.7"
    },
    "outputs": {
    }
  },
  "data": {
    "google_dns_keys": {
      "redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5": {
        "//": {
          "metadata": {
            "path": "airflow-setup/redacted-project-id-airflow-redacted-hash.com/redacted-project-id-airflow-redacted-hash.com-dns-keys",
            "uniqueId": "redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5"
          }
        },
        "managed_zone": "${google_dns_managed_zone.redacted-project-id-airflow-redacted-hashcom.id}"
      }
    },
    …
  },
  "resource": {
    "google_clouddomains_registration": {
      "redacted-project-id-airflow-redacted-hashcom_DE4CED25": {
        "//": {
          "metadata": {
            "path": "airflow-setup/redacted-project-id-airflow-redacted-hash.com/redacted-project-id-airflow-redacted-hash.com",
            "uniqueId": "redacted-project-id-airflow-redacted-hashcom_DE4CED25"
          }
        },
        "contact_settings": {
          "admin_contact": {
            "email": "<redacted>",
            "phone_number": "<redacted>",
            "postal_address": {
              "address_lines": [
                "<redacted>",
                "<redacted>"
              ],
              "administrative_area": "CA",
              "locality": "San Francisco",
              "postal_code": "<redacted>",
              "recipients": [
                "<redacted>"
              ],
              "region_code": "US"
            }
          },
          "privacy": "REDACTED_CONTACT_DATA",
          "registrant_contact": {
            "email": "<redacted>",
            "phone_number": "<redacted>",
            "postal_address": {
              "address_lines": [
                "<redacted>",
                "<redacted>"
              ],
              "administrative_area": "CA",
              "locality": "San Francisco",
              "postal_code": "<redacted>",
              "recipients": [
                "<redacted>"
              ],
              "region_code": "US"
            }
          },
          "technical_contact": {
            "email": "<redacted>",
            "phone_number": "<redacted>",
            "postal_address": {
              "address_lines": [
                "<redacted>",
                "<redacted>"
              ],
              "administrative_area": "CA",
              "locality": "San Francisco",
              "postal_code": "<redacted>",
              "recipients": [
                "<redacted>"
              ],
              "region_code": "US"
            }
          }
        },
        "dns_settings": {
          "custom_dns": {
            "ds_records": [
              {
                "algorithm": "${upper(data.google_dns_keys.redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5.key_signing_keys[0].algorithm)}",
                "digest": "${data.google_dns_keys.redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5.key_signing_keys[0].digests[0].digest}",
                "digest_type": "${upper(data.google_dns_keys.redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5.key_signing_keys[0].digests[0].type)}",
                "key_tag": "${data.google_dns_keys.redacted-project-id-airflow-redacted-hashcom_redacted-project-id-airflow-redacted-hashcom-dns-keys_3076F6A5.key_signing_keys[0].key_tag}"
              }
            ],
            "name_servers": [
              "${element([ for key, val in toset(sort(google_dns_managed_zone.redacted-project-id-airflow-redacted-hashcom.name_servers)): trimsuffix(val, \".\")], 3)}",
              "${element([ for key, val in toset(sort(google_dns_managed_zone.redacted-project-id-airflow-redacted-hashcom.name_servers)): trimsuffix(val, \".\")], 1)}",
              "${element([ for key, val in toset(sort(google_dns_managed_zone.redacted-project-id-airflow-redacted-hashcom.name_servers)): trimsuffix(val, \".\")], 0)}",
              "${element([ for key, val in toset(sort(google_dns_managed_zone.redacted-project-id-airflow-redacted-hashcom.name_servers)): trimsuffix(val, \".\")], 2)}"
            ]
          }
        },
        "domain_name": "redacted-project-id-airflow-redacted-hash.com",
        "labels": {
          "created-by": "${replace(data.terraform_remote_state.cross-stack-reference-input-terraform-deployer.outputs.cross-stack-output-google_project_serviceapi-domains-googleapis-comservice, \".\", \"-\")}",
          "created-by-2": "${replace(replace(data.terraform_remote_state.cross-stack-reference-input-terraform-deployer.outputs.cross-stack-output-google_project_iam_memberterraform-service-account_roles-domainsadmin-redacted-project-id_C8FE6C79member, \"serviceAccount:\", \"\"), \"/[@\\\\.]/\", \"-\")}"
        },
        "location": "global",
        "yearly_price": {
          "currency_code": "USD",
          "units": "12"
        }
      }
    },
    "google_dns_managed_zone": {
      "redacted-project-id-airflow-redacted-hashcom": {
        "//": {
          "metadata": {
            "path": "airflow-setup/redacted-project-id-airflow-redacted-hash.com",
            "uniqueId": "redacted-project-id-airflow-redacted-hashcom"
          }
        },
        "dns_name": "redacted-project-id-airflow-redacted-hash.com.",
        "dnssec_config": {
          "state": "on"
        },
        "labels": {
          "created-by": "${replace(data.terraform_remote_state.cross-stack-reference-input-terraform-deployer.outputs.cross-stack-output-google_project_serviceapi-dns-googleapis-comservice, \".\", \"-\")}",
          "created-by-2": "${replace(replace(data.terraform_remote_state.cross-stack-reference-input-terraform-deployer.outputs.cross-stack-output-google_project_iam_memberterraform-service-account_roles-dnsadmin-redacted-project-id_7BA57828member, \"serviceAccount:\", \"\"), \"/[@\\\\.]/\", \"-\")}"
        },
        "name": "redacted-project-id-airflow-redacted-hash-com"
      }
    },
    …
  },
  "terraform": {
    …
    "required_providers": {
      "google": {
        "source": "google",
        "version": "5.25.0"
      }
    }
  }
}

Debug Output

redacted-airflow-680f608d.com/REDACTED.com) must be replaced
               -/+ resource "google_clouddomains_registration" "REDACTEDcom_DE4CED25" {
                            …
                     ~ dns_settings {
                         ~ custom_dns {
                             ~ name_servers = [ # forces replacement
                                 - "ns-cloud-d4.googledomains.com",
                                 - "ns-cloud-d3.googledomains.com",
                                 - "ns-cloud-d2.googledomains.com",
                                   "ns-cloud-d1.googledomains.com",
                                 + "ns-cloud-d2.googledomains.com",
                                 + "ns-cloud-d3.googledomains.com",
                                 + "ns-cloud-d4.googledomains.com",
                               ]
                               ds_records {
                                   algorithm   = "RSASHA256"
                                   digest      = "9EF0ACBC36F8EBD6A868B7758136675486D0CFD6549FAD29008F00C3CAF9E764"
                                   digest_type = "SHA256"
                                   key_tag     = 14324
                               }
                           }
                       }
                                …
                   }
airflow-setup Error: Error creating Registration: googleapi: Error 400: Domain REDACTED.com is already registered

Expected Behavior

The GCP API keeps reordering the returned order of the name servers. This should not cause replacement of the resource.

Additionally, the resource cannot be recreated easily due to the nature of domains.

Actual Behavior

It tries to delete and recreate the domain registration which fails due to the domain already existing.

Every time this happens, we have to manually edit the tf state to re-add the resource and change the ordering of the name servers to match what the API is now suddenly returning. We then change the terraform code to order the name servers to match the state + GCP API.

This inevitably breaks again and we have to redo this.

Steps to reproduce

create a google_dns_managed_zone resource
create google_clouddomains_registration resource passing the name servers from the created managed zone under dns_settings.
wait for terraform to complain that the domain registration's name servers in GCP no longer matches the terraform state + the generated resource configuration

Important Factoids

We use cdktf python to generate terraform resources. With the following package versions:

cdktf==0.20.7
cdktf-cdktf-provider-google==13.14.0
cdktf-cdktf-provider-google-beta==13.14.0

References

No response

b/356441102

ggtisc commented 1 month ago

Hi @fathom-aniket!

According to the error message REDACTED.com is already registered specifies that a resource with this domain_name was already created and it is not possible to create another with the same name, even if you deleted it from tfstate file.

As you can see in terraform registry is published an alert about this resource, implying that if a resource with the same domain_name was previously created, it will not be deleted but abandoned, which also means that if a domain_name was previously used for any reason, even considering errors, testing, among other possible actions, a new one should be used.

You can read more about it in this link on how to handle these situations.

fathom-aniket commented 1 month ago

Hello @ggtisc and thank you for your response!

According to the error message REDACTED.com is already registered specifies that a resource with this domain_name was already created and it is not possible to create another with the same name, even if you deleted it from tfstate file.

The error message is not the core issue, but is the result of the actual issue (see Actual and Expected Behavior).

The core issue is that:

The GCP API keeps reordering the returned order of the name servers.
This forces a replacement of the existing deployed google_clouddomains_registration resource.
When the cdktf deploy tries to recreate the resource, it fails to properly delete the resource (and just removes it from the terraform state instead), leading to the REDACTED.com is already registered error message on creation.

The expected behavior would be that the GCP API should not return a different order of the name servers or doing so should not trigger a replacement. Please let me know if something is unclear, thanks!

ggtisc commented 1 month ago

Confirmed issue!

In this example we need to use a set instead of a list, but it looks like we need to fix it.

hashicorp / terraform-provider-google