hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.34k stars 1.74k forks source link

Crash error: Root resource was present, but now absent. #8128

Closed chaopeng closed 3 years ago

chaopeng commented 3 years ago

I have see a when "Root resource was present, but now absent." error when upgrade the version of terraform-provider-google to 3.51.0

In this PR https://github.com/terraform-google-modules/terraform-google-healthcare/pull/36

Community Note

Terraform Version

Affected Resource(s)

I don't know yet. It seems related to google_healthcare_dataset

Terraform Configuration Files

https://github.com/terraform-google-modules/terraform-google-healthcare/blob/master/examples/simple_example/main.tf

Debug Output

Panic Output

https://github.com/terraform-google-modules/terraform-google-healthcare/pull/36/checks?check_run_id=1658019589

Error: Provider produced inconsistent result after apply

When applying changes to
module.example.module.healthcare.google_healthcare_dataset.dataset, provider
"registry.terraform.io/hashicorp/google" produced an unexpected new value:
Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

No error

Actual Behavior

Steps to Reproduce

  1. clone the https://github.com/terraform-google-modules/terraform-google-healthcare
  2. create a GCP with pubsub and healthcare API enable
  3. terraform apply

Important Factoids

References

venkykuberan commented 3 years ago

@chaopeng The information you provided isn't adequate enough to understand the issue . We need to know what resource is failing and provide the debug log to help us identify where its filling?

Please also provide the specific resource config or plan output to repro the issue on our end

yeweidaniel commented 3 years ago

Hi, I am also running into this error. With the tf script being:

module "danielye_test4_healthcare_dataset" {
  source  = "terraform-google-modules/healthcare/google"
  version = "~> 1.2.0"

  name     = "danielye-test4-healthcare-dataset"
  project  = module.project.project_id
  location = "us-central1"

  fhir_stores = [
    {
      name    = "intermediate-fhir-store"
      version = "R4"

      enable_update_create          = true
      disable_referential_integrity = true
      notification_config = {
        pubsub_topic = "projects/danielye-test4-pipeline/topics/${module.danielye_test4_intermediate_store_topic.topic}"
      }
    },
    {
      name    = "final-fhir-store"
      version = "R4"

      enable_update_create          = true
      disable_referential_integrity = true
    },
  ]
}

I get the following error:

Error: Provider produced inconsistent result after apply

When applying changes to module.danielye_test4_healthcare_dataset.google_healthcare_dataset.dataset, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

xingao267 commented 3 years ago

I was able to reproduce the same issue with the following configs. My Terraform version is 0.14.4. And I think the same issue occurs with Terraform 0.13 as well.

The project itself matters. If a dataset has been successfully created in a project before, then the issue seems do not occur. So to reproduce the issue, create a new project every time.

terraform {
  required_version = ">=0.14"
  required_providers {
    google      = "~> 3.0"
    google-beta = "~> 3.0"
  }
}

module "project" {
  source                  = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
  name                    = "my-project"
  org_id                  = ""
  folder_id               = "xxxxxx"
  billing_account         = "xxxxxx"
  lien                    = true
  default_service_account = "keep"
  create_project_sa       = false
  activate_apis = [
    "compute.googleapis.com",
    "healthcare.googleapis.com",
  ]
}

module "healthcare_dataset" {
  source  = "terraform-google-modules/healthcare/google"
  version = "~> 1.2.0"

  name     = "healthcare-dataset"
  project  = module.project.project_id
  location = "us-central1"

  fhir_stores = [
    {
      name    = "final-fhir-store"
      version = "R4"

      enable_update_create          = true
      disable_referential_integrity = true
    },
  ]
}
xingao267 commented 3 years ago

I can also reproduce the error with just the google_healthcare_dataset resource. So it should not be an issue in the healthcare module but in the provider. And I forgot to mention that the version of provider I'm using is 3.52.0.

module "project" {
  source                  = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
  name                    = "my-project"
  org_id                  = ""
  folder_id               = "xxx"
  billing_account         = "xxx"
  lien                    = true
  default_service_account = "keep"
  create_project_sa       = false
  activate_apis = [
    "compute.googleapis.com",
    "healthcare.googleapis.com",
  ]
}

resource "google_healthcare_dataset" "default" {
  name     = "healthcare-dataset"
  location = "us-central1"
  project  = module.project.project_id
}
xingao267 commented 3 years ago

My guess is such safety check was introduced in Terraform 0.12 as https://www.terraform.io/docs/extend/terraform-0.12-compatibility.html#inaccurate-plans but not enforced so this resource was fine with Terraform 0.12. However, it's likely since Terraform 0.13, such safety check is enforced and a fix needs to be implemented for the google_healthcare_dataset resource.

xingao267 commented 3 years ago

Logs when I set TF_LOG=DEBUG

2021-01-18T22:25:26.325-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:26 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
POST /v1/projects/xingao-15/locations/us-central1/datasets?alt=json&datasetId=healthcare-dataset HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Length: 30
Content-Type: application/json
Accept-Encoding: gzip

{
 "name": "healthcare-dataset"
}

-----------------------------------------------------: timestamp=2021-01-18T22:25:26.325-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:30 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.CreateDataset"
X-Google-Gfe-Handshake-Trace: GFE: acyyzk8:443,Mentat oracle: [2002:a05:7500:1087:b029:0:c83a:566a]:9801
X-Google-Gfe-Request-Trace: acyyzk8:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
X-Xss-Protection: 0

75
{
-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"
}

0

-----------------------------------------------------: timestamp=2021-01-18T22:25:30.830-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Finished creating Dataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset": map[string]interface {}{"name":"projects/xingao-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"}: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Waiting for state to become: [success]: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: request attempt 0: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.832-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/xingao-15/locations/us-central1/datasets/healthcare-dataset?alt=json HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Type: application/json
Accept-Encoding: gzip

-----------------------------------------------------: timestamp=2021-01-18T22:25:30.832-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
race.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0

1bc
{
  "error": {
    "code": 404,
    "message": "dataset not initialized",
    "status": "NOT_FOUND",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
      }
    ]
  }
}

0

-----------------------------------------------------: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
fish-Status: CA0gBEBG
X-Google-Trace: http://trace.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0

1bc
{
  "error": {
    "code": 404,
    "message": "dataset not initialized",
    "status": "NOT_FOUND",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
      }
    ]
  }
}

0
: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.043-0500 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [WARN] Removing HealthcareDataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset" because it's gone: timestamp=2021-01-18T22:25:31.042-0500
2021/01/18 22:25:31 [DEBUG] google_healthcare_dataset.default: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to google_healthcare_dataset.default, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Error: Provider produced inconsistent result after apply

When applying changes to google_healthcare_dataset.default, provider
"registry.terraform.io/hashicorp/google" produced an unexpected new value:
Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

2021-01-18T22:25:31.056-0500 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/3.52.0/linux_amd64/terraform-provider-google_v3.52.0_x5 pid=2149476
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin exited
xingao267 commented 3 years ago

The dataset was actually created successfully, but Terraform's GET call to verify it's created failed due to dataset not initialized . I guess some sort of retry logic should be implemented to retry the GET request so it doesn't fail and just exit in the first attempt.

xingao267 commented 3 years ago

@venkykuberan @rileykarson have you seen similar issue before? I can take a stab in fixing this, but might need some guidance from you or existing examples for how to add retry logics for a resource in magic-module.

xingao267 commented 3 years ago

Browsing the magic module repo and found something like https://github.com/GoogleCloudPlatform/magic-modules/blob/d910e58b30b67e1c641463ab56de084ed4351225/products/pubsub/terraform.yaml#L32 plus https://github.com/GoogleCloudPlatform/magic-modules/blob/master/third_party/terraform/utils/error_retry_predicates.go. Trying something similar for healthcare dataset.

xingao267 commented 3 years ago

Worked. Sent https://github.com/GoogleCloudPlatform/magic-modules/pull/4412

xingao267 commented 3 years ago

The fix is planned to be released in google provider 3.54.0 next Monday.

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!