Provider produced inconsistent result after apply

ocofaigh commented 2 years ago

TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: ╷
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ Error: Provider produced inconsistent result after apply
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ 
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ When applying changes to ibm_iam_service_id.iam_secret_generator, provider
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ "provider[\"registry.terraform.io/ibm-cloud/ibm\"]" produced an unexpected
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ new value: Root resource was present, but now absent.
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ 
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ This is a bug in the provider, which should be reported in the provider's
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ own issue tracker.
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: ╵
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: time=2022-02-09T06:21:20Z level=error msg=Module /workspace/goldeneye-input-repo/scaffold/dev-au-syd/control-plane/account/secrets-manager has finished with an error: 1 error occurred:
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66:    * exit status 1
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: 
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66:  prefix=[/workspace/goldeneye-input-repo/scaffold/dev-au-syd/control-plane/account/secrets-manager]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

ibm provider version: v1.38.1 TF version: 1.1.4

Affected Resource(s)

ibm_iam_service_id

Terraform Configuration Files

resource "ibm_iam_service_id" "iam_secret_generator" {
  name        = "sid:0.0.1:${var.secrets_manager_name}-iam-secret-generator:automated:simple-service:secret-manager:"
  description = "ServiceID that can generate IAM ServiceID API Keys stored in Secrets Manager secrets"
}

Debug Output

scaffolding-terragrunt-test-logs-220209-065911.tar.gz

Panic Output

Expected Behavior

No error

Actual Behavior

Error (see top of description)

Steps to Reproduce

terraform apply

Important Factoids

References

0000

kavya498 commented 2 years ago

@ocofaigh , This looks like an intermittent issue from IAM API..

The below is seen when id that is set to the resource is again set "" during resource creation.. As I suspected, the same thing is happening in your case

When applying changes to ibm_iam_service_id.iam_secret_generator, provider
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ "provider[\"registry.terraform.io/ibm-cloud/ibm\"]" produced an unexpected
TestTerragruntScaffoldingDevAuSyd 2022-02-09T06:21:20Z logger.go:66: │ new value: Root resource was present, but now absent.

Provider flow:

POST call to serviceID
sets id to resource
returns read where it does GET call and sets all arguments to state.. Here in READ, if GET gives 404 we set ID to "" assuming since resource doesnot exist terraform should recreate one.. but this scenario should not occur on creating new resource..

This looks like an intermittent issue from IAM.. Looking into your logs.. POST has happend and it has created service ID but GET gives 404.. Can you please Reach out to IAM Identity team regarding this wierd behaviour?

POST /v1/serviceids/ HTTP/1.1
Host: iam.cloud.ibm.com
User-Agent: platform-services-go-sdk/0.22.6 (lang=go; arch=amd64; os=linux; go.version=go1.16.13)
Content-Length: 271
Accept: application/json
Authorization: [redacted]
Content-Type: application/json
X-Original-User-Agent: terraform-provider-ibm/1.38.1
Accept-Encoding: gzip

{"account_id":"abac0df06b644a9cabc6e44f55b3880e","description":"ServiceID that can generate IAM ServiceID API Keys stored in Secrets Manager secrets","name":"sid:0.0.1:tg-scaffoldtest-xf8ks5-secrets-manager-iam-secret-generator:automated:simple-service:secret-manager:"}: timestamp=2022-02-09T06:55:03.433Z

...

HTTP/1.1 201 Created
Content-Length: 689
Akamai-Grn: 0.dd5dda17.1644387085.60c89c1
Cache-Control: no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Language: en-US
Content-Type: application/json
Date: Wed, 09 Feb 2022 06:11:28 GMT
Etag: 1-323d6871d2a38193dceedd2fa7b098dd
Expires: 0
Pragma: no-cache
Set-Cookie: sessioncookie="101ad0d4d81570ff"; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: aWFtaWQtNy4xLTEyNzMwLWU1NDc2ZjgtNzhkNDVjZGRiZC1oazRsbg-3f072a1f5efa47e181cf4760925fbd09
X-Content-Type-Options: nosniff
X-Proxy-Upstream-Service-Time: 3447

{"id":"ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8","iam_id":"iam-ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8","entity_tag":"1-323d6871d2a38193dceedd2fa7b098dd","crn":"crn:v1:bluemix:public:iam-identity::a/abac0df06b644a9cabc6e44f55b3880e::serviceid:ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8","locked":false,"created_at":"2022-02-09T06:11+0000","modified_at":"2022-02-09T06:11+0000","account_id":"abac0df06b644a9cabc6e44f55b3880e","name":"sid:0.0.1:tg-scaffoldtest-xf8ks5-secrets-manager-iam-secret-generator:automated:simple-service:secret-manager:","description":"ServiceID that can generate IAM ServiceID API Keys stored in Secrets Manager secrets","unique_instance_crns":[]}: timestamp=2022-02-09T06:11:28.801Z
2022-02-09T06:11:28.802Z [INFO]  provider.terraform-provider-ibm_v1.38.1: 2022/02/09 06:11:28 [Debug] Request:

GET /v1/serviceids/ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8 HTTP/1.1
Host: iam.cloud.ibm.com
User-Agent: platform-services-go-sdk/0.22.6 (lang=go; arch=amd64; os=linux; go.version=go1.16.13)
Accept: application/json
Authorization: [redacted]
X-Original-User-Agent: terraform-provider-ibm/1.38.1
Accept-Encoding: gzip

: timestamp=2022-02-09T06:11:28.801Z
2022-02-09T06:11:28.802Z [INFO]  provider.terraform-provider-ibm_v1.38.1: 2022/02/09 06:11:28 [DEBUG] GET https://iam.cloud.ibm.com/v1/serviceids/ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8: timestamp=2022-02-09T06:11:28.801Z
2022-02-09T06:11:28.843Z [DEBUG] provider: using plugin: version=5
2022-02-09T06:11:28.843Z [DEBUG] provider.terraform-provider-null_v3.1.0_x5: plugin address: address=/tmp/plugin231293451 network=unix timestamp=2022-02-09T06:11:28.843Z
2022-02-09T06:11:28.906Z [TRACE] BuiltinEvalContext: Initialized "provider[\"registry.terraform.io/hashicorp/null\"]" provider for provider["registry.terraform.io/hashicorp/null"]
2022-02-09T06:11:28.906Z [TRACE] NodeApplyableProvider: validating configuration for provider["registry.terraform.io/hashicorp/null"]
2022-02-09T06:11:28.906Z [TRACE] provider.stdio: waiting for stdio data
2022-02-09T06:11:28.907Z [DEBUG] created provider logger: level=trace
2022-02-09T06:11:28.907Z [INFO]  provider: configuring client automatic mTLS
2022-02-09T06:11:28.932Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/ibm-cloud/ibm/1.38.1/linux_amd64/terraform-provider-ibm_v1.38.1 args=[.terraform/providers/registry.terraform.io/ibm-cloud/ibm/1.38.1/linux_amd64/terraform-provider-ibm_v1.38.1]
2022-02-09T06:11:28.941Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/ibm-cloud/ibm/1.38.1/linux_amd64/terraform-provider-ibm_v1.38.1 pid=4647
2022-02-09T06:11:28.942Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/ibm-cloud/ibm/1.38.1/linux_amd64/terraform-provider-ibm_v1.38.1
2022-02-09T06:11:29.031Z [DEBUG] provider.terraform-provider-ibm_v1.38.1: 2022/02/09 06:11:29 IBM Cloud Provider version 1.38.1  
2022-02-09T06:11:29.049Z [INFO]  provider.terraform-provider-ibm_v1.38.1: configuring server automatic mTLS: timestamp=2022-02-09T06:11:29.049Z
2022-02-09T06:11:29.052Z [INFO]  provider.terraform-provider-ibm_v1.38.1: 2022/02/09 06:11:29 [Debug] Response:
HTTP/1.1 404 Not Found
Content-Length: 314
Akamai-Grn: 0.dd5dda17.1644387088.60c8cb6
Cache-Control: no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Language: en-US
Content-Type: application/json
Date: Wed, 09 Feb 2022 06:11:29 GMT
Expires: 0
Pragma: no-cache
Set-Cookie: sessioncookie="553642c11d303773"; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transaction-Id: aWFtaWQtNy4xLTEyNzMwLWU1NDc2ZjgtNzhkNDVjZGRiZC1xNjdkcg-94c8803036214c0b908f58c5199aa8f3
X-Content-Type-Options: nosniff
X-Proxy-Upstream-Service-Time: 203

{"errors":[{"code":"not_found","message_code":"BXNIM0102E","message":"Unable to find object","details":"Object type 'ServiceId' with ID 'ServiceId-52b97c5d-e7c6-424c-a031-dbfc27d0caa8' not found"}],"trace":"aWFtaWQtNy4xLTEyNzMwLWU1NDc2ZjgtNzhkNDVjZGRiZC1xNjdkcg-94c8803036214c0b908f58c5199aa8f3","status_code":404}: timestamp=2022-02-09T06:11:29.051Z

ocofaigh commented 2 years ago

@kavya498 Thank you for checking. I have reported to IAM here.

It looks like the POST responded with 201 at Wed, 09 Feb 2022 06:11:28 And then the GET was got 404 at Wed, 09 Feb 2022 06:11:29 GMT

So just 1 second later - So it is more than likely a timing issue. How about adding some retries to the GET call in the provider code?

kavya498 commented 2 years ago

Yeah, But, I feel this timing issues has to be reolved by APIs, Atleast some kind of status has to be given by GET call instead of 404

ocofaigh commented 2 years ago

@kavya498 Feel free to join the discussions at https://ibm-cloudplatform.slack.com/archives/C3C46LY7N/p1644575432757299

ocofaigh commented 2 years ago

@kavya498 I think I hit another issue due to the same root cause. Perhaps you can confirm. If it is, then as suggested by IAM, perhaps we need to handle this in the provider code...

From the logs, it looks like while calling ibm_iam_authorization_policy.policy RC returned 201:

2022-02-16T12:21:49.919Z [INFO] provider.terraform-provider-ibm_v1.38.2: 2022/02/16 12:21:49 [Debug] Response: HTTP/1.1 201 Created

But then the subsequent GET failed with 404:

2022-02-16T12:21:49.920Z [INFO]  provider.terraform-provider-ibm_v1.38.2: 2022/02/16 12:21:49 [DEBUG] GET https://iam.cloud.ibm.com/v1/policies/612302ad-8de4-4fc5-8b61-122fd5b0723b: timestamp=2022-02-16T12:21:49.919Z
2022-02-16T12:21:50.049Z [INFO]  provider.terraform-provider-ibm_v1.38.2: 2022/02/16 12:21:50 [Debug] Response:
HTTP/1.1 404 Not Found

Full trace: trace.log

IBM-Cloud / terraform-provider-ibm