[Provisioner] CBR rules intermittently fail to create

shemau commented 8 months ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

Various terraform v1.5.7 with IBM provider 1.57.0 through 1.59.0

Affected Resource(s)

ibm_cbr_rule

Terraform Configuration Files

CBR created for example for a COS bucket

Debug Output

TestRunFSCloudExample 2023-11-16T06:11:28Z logger.go:66: module.cos_fscloud.module.buckets.module.buckets["cos-fscloud-tqdg56-primary-bucket"].module.bucket_cbr_rule[0].ibm_cbr_rule.cbr_rule: Creating...
...
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: module.cos_fscloud.module.buckets.module.buckets["cos-fscloud-tqdg56-primary-bucket"].time_sleep.wait_for_authorization_policy: Creation complete after 30s [id=2023-11-16T06:11:55Z]
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: ╷
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ Error: Provider produced inconsistent result after apply
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ 
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ When applying changes to
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ module.cos_fscloud.module.buckets.module.buckets["cos-fscloud-tqdg56-primary-bucket"].module.bucket_cbr_rule[0].ibm_cbr_rule.cbr_rule,
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ provider "provider[\"[registry.terraform.io/ibm-cloud/ibm\"]"](http://registry.terraform.io/ibm-cloud/ibm/%22]%22) produced an
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ unexpected new value: Root resource was present, but now absent.
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ 
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ This is a bug in the provider, which should be reported in the provider's
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: │ own issue tracker.
TestRunFSCloudExample 2023-11-16T06:11:55Z logger.go:66: ╵
TestRunFSCloudExample 2023-11-16T06:11:55Z retry.go:144: 'terraform [apply -input=false -auto-approve -var bucket_hpcs_key_crn=crn:v1:bluemix:public:hs-crypto:us-south:a/<account_id>:<service_id>:key:<key_id> -var prefix=cos-fscloud-tqdg56 -var region=us-south -var resource_group=geretain-test-cos-base -var resource_tags=[] -var existing_at_instance_crn=crn:v1:bluemix:public:logdnaat:eu-de:a/<account_id>:<service_id>:: -var access_tags=["geretain-dev:permanent-test-tag-1", "geretain-dev-1:permanent-test-tag-4"] -var bucket_existing_hpcs_instance_guid=<service_id> -lock=false]' failed with the error 'error while running command: exit status 1; ╷
│ Error: Provider produced inconsistent result after apply

Indicates that the CBR rule was being created (no completed message) when the error was created.

Panic Output

Expected Behavior

The CBR rule should be created. This is an intermittent error, most times it just works.

Actual Behavior

Occasionally the CBR rule fails to create.

Steps to Reproduce

terraform apply

Important Factoids

The issue is intermittent

References

shemau commented 8 months ago

This occurred again, this time with provider 1.59.1.

shemau commented 7 months ago

This occurred again today, this time the provider was 1.60.0

akocbek commented 5 months ago

This occurred again, this time the provider was 1.61.0

shemau commented 4 months ago

This is still happening with provider 1.62.0

shemau commented 2 months ago

This is still happening with the provider 1.65.0

TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: module.cos_bucket3.module.bucket_cbr_rule[0].ibm_cbr_rule.cbr_rule: Creation complete after 1s [id=4d68e8a95dd74d26dda75ded8c15855c]
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: ╷
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ Error: Provider produced inconsistent result after apply
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ 
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ When applying changes to
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ module.cos_bucket2.module.bucket_cbr_rule[0].ibm_cbr_rule.cbr_rule,
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ provider "provider[\"[registry.terraform.io/ibm-cloud/ibm\"]"](http://registry.terraform.io/ibm-cloud/ibm/%22]%22) produced an
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ unexpected new value: Root resource was present, but now absent.
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ 
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ This is a bug in the provider, which should be reported in the provider's
TestRunAdvancedExample 2024-05-16T05:11:02Z logger.go:66: │ own issue tracker.

@hkantare can we get this assigned to CBR team? It is a intermittent, but persistent error that routinely causes PR and CI pipeline failures. I am pretty sure that the issue is that the CBR has been created, but is temporarily unavailable at the time the provider checks if it exists. This causes the create to fail and the resource to be missing from the state file. Unfortunately on a re-run then finds that the resource does exist, so it needs to be manually imported (or destroyed from the command line) before an apply can be run again.

IBM-Cloud / terraform-provider-ibm