Confusing error messages when creating encrypted EBS volumes

[acdha]

Currently aws_ebs_volume can fail in a confusing way when using a KMS CMK to encrypt the volume, “successfully” creating the volume but having attachments fail and no sign of it in console or CLI.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.24
+ provider.aws v2.53.0
+ provider.external v1.2.0
+ provider.http v1.1.1

Affected Resource(s)

• aws_ebs_volume

Terraform Configuration Files

resource "aws_ebs_volume" "sgw_cache" {
…
  encrypted         = true
  kms_key_id        = aws_kms_key.app_storage.arn
}

Expected Behavior

Encrypted volume was created

Actual Behavior

Encrypted volume was displayed by Terraform as created but was not actually created and other operations such as attachments will fail with InvalidVolume.NotFound.

This is because creating an EBS volume with a CMK is actually a multi-stage operation as described in https://docs.aws.amazon.com/kms/latest/developerguide/services-ebs.html#ebs-cmk and the API doesn't do full validation. What happens is that the initial API call returns success but immediately fails as soon as it hits the step where it makes the KMS GenerateDataKeyWithoutPlaintext call. The volume ID is silently deleted at that point (without even a record in CloudTrail) so the returned volume ID which Terraform uses everywhere else will trigger those NotFound exceptions.

Ideally, this would trigger a validation error but I'm suspect there are people running Terraform in a context where it doesn't have permissions to retrieve the KMS policy so perhaps this could be some help text in the error handler if the volume's kms_key_id points to a non-AWS KMS or at least a note in the documentation.

Steps to Reproduce

1. terraform apply

[aSapien]

Still reproducing with:

Terraform v0.15.0
provider hashicorp/aws v3.20.0
aws-cli/2.1.30

Thanks for documenting!

[aSapien]

Checking the CloudTrail Event history, I noticed that the user performing the operation on kms.amazonaws.com (MFA protected resource) had the attribute "mfaAuthenticated": "false", which caused a "errorCode": "AccessDenied".

Authenticating with MFA again solved the KMS issue for me.

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!