Failed ES domain upgrade error isn't helpful

tomelliff commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.10

provider.aws v2.33.99 (my fork based on https://github.com/terraform-providers/terraform-provider-aws/pull/6314 which is just that one ECS related change on top of 2.33.0)

Affected Resource(s)

aws_elasticsearch_domain

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

Debug Output

The relevant part of the debug log is small so posting it directly here:

2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: 2019/11/28 22:56:01 [DEBUG] [aws-sdk-go] DEBUG: Response es/GetUpgradeStatus Details:
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: ---[ RESPONSE ]--------------------------------------
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: HTTP/1.1 200 OK
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: Connection: close
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: Content-Length: 97
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: Content-Type: application/json
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: Date: Thu, 28 Nov 2019 22:56:00 GMT
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: X-Amzn-Requestid: 3f850bbc-1232-11ea-bc06-1fdf099cbf0b
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: 
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: 
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: -----------------------------------------------------
2019-11-28T22:56:01.273Z [DEBUG] plugin.terraform-provider-aws_v2.33.99_x4: 2019/11/28 22:56:01 [DEBUG] [aws-sdk-go] {"StepStatus":"FAILED","UpgradeName":"Upgrade from 6.8 to 7.1","UpgradeStep":"PRE_UPGRADE_CHECK"}
2019/11/28 22:56:01 [DEBUG] module.elasticsearch.aws_elasticsearch_domain.elasticsearch: apply errored, but we're indicating that via the Error pointer rather than returning it: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

Expected Behavior

My cluster is failing the upgrade eligibility checks but I'd expect to see the error correctly reported by Terraform with something like the following:

Cluster has 1160.0 shards per node which exceeds the setting cluster.max_shards_per_node value 1000

Actual Behavior

Error: unexpected state 'FAILED', wanted target 'SUCCEEDED'. last error: %!s(<nil>)

Steps to Reproduce

Get an ES cluster in a position that it can't be upgraded for whatever reason
Set the Terraform config to upgrade the version via a valid in place upgrade path
terraform apply

Important Factoids

I've moved from a 2 AZ ES cluster to a 3 AZ ES cluster in place and then immediately moved to 6.8 and then attempted to again upgrade to 7.2 but this is causing the above error on the AWS side. That bit is fine but I'd expect Terraform to properly show the error instead of %!s(<nil>)

I wrote this in place upgrade code but didn't have a good way of inducing an upgrade failure so couldn't really test what happened in that case but it looks like AWS's API doesn't return an error, just the FAILED StepStatus field. The GetUpgradeHistory API endpoint will show the results of any attempted upgrades in reverse chronological order so it's possible we could retrieve the first failed result from that for the domain and return the list of UpgradeStepItem.Issues.

I am wary that I don't know a good way to force an ES cluster into a bad state though so this might be tricky to test once my ES cluster is back in to a good place.

References

https://github.com/terraform-providers/terraform-provider-aws/pull/6243

justinretzolk commented 2 years ago

Hey @tomelliff 👋 Thank you for taking the time to file this issue! Given that there's been a number of AWS provider releases since you initially filed it, can you confirm whether you're still experiencing this behavior?