[Bug]: Opensearch Serverless Collection creation fails with "missing" encryption policy

[cornevandyk]

Terraform Core Version

1.9.2

AWS Provider Version

5.59

Affected Resource(s)

aws_opensearchserverless_collection aws_opensearchserverless_security_policy

Expected Behavior

The opensearch serverless collection should be created, along with all defined security policies.

Actual Behavior

The collection creation fails, with an error that a required security policy of type "encryption" is missing, although the security policy was already created.

Re-running the apply succeeds, since the security policy is already there. Adding a depends_on to aws_opensearchserverless_collection also works, since it apparently forces the collection creation to wait for the aws_opensearchserverless_security_policy resource, but this should not be required, as the security policy was created before the collection.

Relevant Error/Panic Output Snippet

aws_opensearchserverless_security_policy.xxxx_xxxx_encryption_policy: Creating...
aws_opensearchserverless_collection.xxxx_xxxx_aoss_collection: Creating...
aws_opensearchserverless_security_policy.xxxx_xxxx_encryption_policy: Creation complete after 0s [id=xxxx-xxxx-encryption-policy]
╷
│ Error: creating AWS OpenSearch Serverless Collection (aacorne-xxxx)
│
│   with aws_opensearchserverless_collection.xxxx_xxxx_aoss_collection,
│   on opensearch.tf line 12, in resource "aws_opensearchserverless_collection" "xxxx_xxxx_aoss_collection":
│   12: resource "aws_opensearchserverless_collection" "xxxx_xxxx_aoss_collection" {
│
│ operation error OpenSearchServerless: CreateCollection, https response error StatusCode: 400, RequestID:
│ 099245a4-5558-46bb-85ad-a0ebb6031e8d, ValidationException: No matching security policy of encryption type found
│ for collection name: aacorne-xxxx. Please create security policy of encryption type for this collection.

Terraform Configuration Files

Only including relevant resources here:

# OpenSearch Serverless collection
resource "aws_opensearchserverless_collection" "xxxx_xxxx_aoss_collection" {
  name = var.xxxx_aoss_collection
  tags = local.tags
}

# AOSS encryption policy, use AWS managed key
resource "aws_opensearchserverless_security_policy" "xxxx_xxxx_encryption_policy" {
  name        = "xxxx-xxxx-encryption-policy"
  type        = "encryption"
  description = "encryption security policy for xxxx-xxxx collection"
  policy = jsonencode({
    Rules = [
      {
        Resource = [
          "collection/${var.xxxx_aoss_collection}"
        ],
        ResourceType = "collection"
      }
    ],
    AWSOwnedKey = true
  })
}

Steps to Reproduce

Plan and apply the given Terraform code.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @cornevandyk 👋 Thank you for taking the time to raise this! In this case, the error is coming from the AWS API itself. We can see that in the logs in the 400 error found here:

│ operation error OpenSearchServerless: CreateCollection, https response error StatusCode: 400, RequestID:
│ 099245a4-5558-46bb-85ad-a0ebb6031e8d, ValidationException: No matching security policy of encryption type found

The collection creation fails, with an error that a required security policy of type "encryption" is missing, although the security policy was already created.

Reviewing the supplied logs, this isn't quite accurate. These three lines indicate that the aws_opensearchserverless_security_policy and aws_opensearchserverless_collection resources are being created in parallel, with the aws_opensearchserverless_security_policy completing just after the aws_opensearchserverless_collection creation begins.

aws_opensearchserverless_security_policy.xxxx_xxxx_encryption_policy: Creating...
aws_opensearchserverless_collection.xxxx_xxxx_aoss_collection: Creating...
aws_opensearchserverless_security_policy.xxxx_xxxx_encryption_policy: Creation complete after 0s [id=xxxx-xxxx-encryption-policy]

In cases such as this, where there's not an implied dependency, using depends_on to create an explicit dependency would be the best path forward. Since you've mentioned that worked, and this is otherwise behaving as I'd expect, I'm going to close this issue. If you encounter any other unexpected behavior in the future, please do let us know!

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.