Closed chaopeng closed 3 years ago
@chaopeng The information you provided isn't adequate enough to understand the issue . We need to know what resource is failing and provide the debug log to help us identify where its filling?
Please also provide the specific resource config or plan output to repro the issue on our end
Hi, I am also running into this error. With the tf script being:
module "danielye_test4_healthcare_dataset" {
source = "terraform-google-modules/healthcare/google"
version = "~> 1.2.0"
name = "danielye-test4-healthcare-dataset"
project = module.project.project_id
location = "us-central1"
fhir_stores = [
{
name = "intermediate-fhir-store"
version = "R4"
enable_update_create = true
disable_referential_integrity = true
notification_config = {
pubsub_topic = "projects/danielye-test4-pipeline/topics/${module.danielye_test4_intermediate_store_topic.topic}"
}
},
{
name = "final-fhir-store"
version = "R4"
enable_update_create = true
disable_referential_integrity = true
},
]
}
I get the following error:
Error: Provider produced inconsistent result after apply
When applying changes to module.danielye_test4_healthcare_dataset.google_healthcare_dataset.dataset, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own issue tracker.
I was able to reproduce the same issue with the following configs. My Terraform version is 0.14.4. And I think the same issue occurs with Terraform 0.13 as well.
The project itself matters. If a dataset has been successfully created in a project before, then the issue seems do not occur. So to reproduce the issue, create a new project every time.
terraform {
required_version = ">=0.14"
required_providers {
google = "~> 3.0"
google-beta = "~> 3.0"
}
}
module "project" {
source = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
name = "my-project"
org_id = ""
folder_id = "xxxxxx"
billing_account = "xxxxxx"
lien = true
default_service_account = "keep"
create_project_sa = false
activate_apis = [
"compute.googleapis.com",
"healthcare.googleapis.com",
]
}
module "healthcare_dataset" {
source = "terraform-google-modules/healthcare/google"
version = "~> 1.2.0"
name = "healthcare-dataset"
project = module.project.project_id
location = "us-central1"
fhir_stores = [
{
name = "final-fhir-store"
version = "R4"
enable_update_create = true
disable_referential_integrity = true
},
]
}
I can also reproduce the error with just the google_healthcare_dataset
resource. So it should not be an issue in the healthcare module but in the provider. And I forgot to mention that the version of provider I'm using is 3.52.0.
module "project" {
source = "github.com/terraform-google-modules/terraform-google-project-factory?ref=c41ba360a6bc6800a30d284b8fa23eb3ef5a8d7f"
name = "my-project"
org_id = ""
folder_id = "xxx"
billing_account = "xxx"
lien = true
default_service_account = "keep"
create_project_sa = false
activate_apis = [
"compute.googleapis.com",
"healthcare.googleapis.com",
]
}
resource "google_healthcare_dataset" "default" {
name = "healthcare-dataset"
location = "us-central1"
project = module.project.project_id
}
My guess is such safety check was introduced in Terraform 0.12 as https://www.terraform.io/docs/extend/terraform-0.12-compatibility.html#inaccurate-plans but not enforced so this resource was fine with Terraform 0.12. However, it's likely since Terraform 0.13, such safety check is enforced and a fix needs to be implemented for the google_healthcare_dataset
resource.
Logs when I set TF_LOG=DEBUG
2021-01-18T22:25:26.325-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:26 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
POST /v1/projects/xingao-15/locations/us-central1/datasets?alt=json&datasetId=healthcare-dataset HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Length: 30
Content-Type: application/json
Accept-Encoding: gzip
{
"name": "healthcare-dataset"
}
-----------------------------------------------------: timestamp=2021-01-18T22:25:26.325-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:30 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.CreateDataset"
X-Google-Gfe-Handshake-Trace: GFE: acyyzk8:443,Mentat oracle: [2002:a05:7500:1087:b029:0:c83a:566a]:9801
X-Google-Gfe-Request-Trace: acyyzk8:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2.esf,acyyzk8:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/2:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
X-Xss-Protection: 0
75
{
-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"
}
0
-----------------------------------------------------: timestamp=2021-01-18T22:25:30.830-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Finished creating Dataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset": map[string]interface {}{"name":"projects/xingao-15/locations/us-central1/datasets/healthcare-dataset/operations/14286000718916616193"}: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Waiting for state to become: [success]: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.831-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Retry Transport: request attempt 0: timestamp=2021-01-18T22:25:30.831-0500
2021-01-18T22:25:30.832-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:30 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/projects/xingao-15/locations/us-central1/datasets/healthcare-dataset?alt=json HTTP/1.1
Host: healthcare.googleapis.com
User-Agent: Terraform/0.14.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.4.0 terraform-provider-google/dev
Content-Type: application/json
Accept-Encoding: gzip
-----------------------------------------------------: timestamp=2021-01-18T22:25:30.832-0500
2021-01-18T22:25:31.042-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
X-Google-Shellfish-Status: CA0gBEBG
race.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0
1bc
{
"error": {
"code": 404,
"message": "dataset not initialized",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
}
]
}
}
0
-----------------------------------------------------: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/1.1 404 Not Found
Connection: close
Transfer-Encoding: chunked
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 19 Jan 2021 03:25:31 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Google-Backends: [2002:a05:663c:1056:b029:120:df74:ce0a]:4001,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Dos-Service-Trace: main:cloud-healthcare-prod-us-central1-api
X-Google-Esf-Cloud-Client-Params: backend_service_name: "healthcare.googleapis.com" backend_fully_qualified_method: "google.cloud.healthcare.v1.dataset.DatasetService.GetDataset" is_dapper_traced: true
X-Google-Gfe-Handshake-Trace: GFE: acyyzk16:443,Mentat oracle: [2002:a05:7500:1109:b029:1:5f0c:f0d3]:9801
X-Google-Gfe-Request-Trace: acyyzk16:443,/bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1.esf,acyyzk16:443
X-Google-Gfe-Response-Body-Transformations: chunked
X-Google-Gfe-Response-Code-Details-Trace: response_code_set_by_backend
X-Google-Gfe-Service-Trace: cloud-healthcare-prod-us-central1-api
X-Google-Netmon-Label: /bns/jc/borg/jc/bns/cloud-healthcare/prod-us-central1.cloud-healthcare-esf/1:esf
X-Google-Security-Signals: FRAMEWORK=HTTPSERVER2
X-Google-Service: cloud-healthcare-prod-us-central1-api
X-Google-Session-Info: CJbH2orbBRoEGBAoCzpsEhloZWFsdGhjYXJlLmdvb2dsZWFwaXMuY29tGIrIz7ieFiJINzY0MDg2MDUxODUwLTZxcjRwNmdwaTZobjUwNnB0OGVqdXE4M2RpMzQxaHVyLmFwcHMuZ29vZ2xldXNlcmNvbnRlbnQuY29tSg86DTEvLzA0YmlKNFAxU36yAQ9LTGVicmRIVHFfT2YzZ0U
fish-Status: CA0gBEBG
X-Google-Trace: http://trace.corp.google.com/trace?host=jcts15-20020a306acf0000b02901292ad7646a&trace_id=0x12a8c88f3e4636cf&id=0xff8fd67c4397534b&start=2021-01-18_19:25:30&rpc_duration=0.041417
X-Xss-Protection: 0
1bc
{
"error": {
"code": 404,
"message": "dataset not initialized",
"status": "NOT_FOUND",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"detail": "[ORIGINAL ERROR] generic::not_found: dataset not initialized, cause (internal only): generic::not_found: dataset not initialized [google.rpc.error_details_ext] { message: \"dataset not initialized\" } 318741215: \"\""
}
]
}
}
0
: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.042-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-01-18T22:25:31.042-0500
2021-01-18T22:25:31.043-0500 [INFO] plugin.terraform-provider-google_v3.52.0_x5: 2021/01/18 22:25:31 [WARN] Removing HealthcareDataset "projects/xingao-15/locations/us-central1/datasets/healthcare-dataset" because it's gone: timestamp=2021-01-18T22:25:31.042-0500
2021/01/18 22:25:31 [DEBUG] google_healthcare_dataset.default: apply errored, but we're indicating that via the Error pointer rather than returning it: Provider produced inconsistent result after apply: When applying changes to google_healthcare_dataset.default, provider "registry.terraform.io/hashicorp/google" produced an unexpected new value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own issue tracker.
Error: Provider produced inconsistent result after apply
When applying changes to google_healthcare_dataset.default, provider
"registry.terraform.io/hashicorp/google" produced an unexpected new value:
Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
2021-01-18T22:25:31.056-0500 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/3.52.0/linux_amd64/terraform-provider-google_v3.52.0_x5 pid=2149476
2021-01-18T22:25:31.059-0500 [DEBUG] plugin: plugin exited
The dataset was actually created successfully, but Terraform's GET call to verify it's created failed due to dataset not initialized . I guess some sort of retry logic should be implemented to retry the GET request so it doesn't fail and just exit in the first attempt.
@venkykuberan @rileykarson have you seen similar issue before? I can take a stab in fixing this, but might need some guidance from you or existing examples for how to add retry logics for a resource in magic-module.
Browsing the magic module repo and found something like https://github.com/GoogleCloudPlatform/magic-modules/blob/d910e58b30b67e1c641463ab56de084ed4351225/products/pubsub/terraform.yaml#L32 plus https://github.com/GoogleCloudPlatform/magic-modules/blob/master/third_party/terraform/utils/error_retry_predicates.go. Trying something similar for healthcare dataset.
The fix is planned to be released in google provider 3.54.0 next Monday.
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!
I have see a when "Root resource was present, but now absent." error when upgrade the version of terraform-provider-google to 3.51.0
In this PR https://github.com/terraform-google-modules/terraform-google-healthcare/pull/36
Community Note
modular-magician
user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot
, a community member has claimed the issue already.Terraform Version
Affected Resource(s)
I don't know yet. It seems related to
google_healthcare_dataset
Terraform Configuration Files
https://github.com/terraform-google-modules/terraform-google-healthcare/blob/master/examples/simple_example/main.tf
Debug Output
Panic Output
https://github.com/terraform-google-modules/terraform-google-healthcare/pull/36/checks?check_run_id=1658019589
Expected Behavior
No error
Actual Behavior
Steps to Reproduce
terraform apply
Important Factoids
References