[BUG] opensearch_snapshot_repository throws "path is not accessible on cluster-manager node"

[fmartinou]

What is the bug?

On an AWS Opensearch cluster (single node without dedicated masters and fine-grained access control enabled), the creation of an opensearch_snapshot_repository pointing to an S3 bucket fails with the following error:

elastic: Error 500 (Internal Server Error): [test] path  is not accessible on cluster-manager node [type=repository_verification_exception]

To be noticed:

• the repository is still properly created on the Opensearch cluster
• the repository is full usable; I can take and restore snapshots on it without any errors
• I can see the files being written to the S3 bucket
• When I execute Terraform a 2nd time, this error don't occur anymore

How can one reproduce the bug?

1. Deploy a managed AWS Opensearch cluster (version 2.11, 1 data node without dedicated masters and fine-grained access control enabled)
2. Create a Snapshot repository using the Opensearch Terraform provider

   resource "opensearch_snapshot_repository" "snapshots" {
   name     = "test"
   type     = "s3"
   settings = {
   bucket   = "my-bucket"
   region   = "eu-west-3"
   role_arn = "a role arn with the policies detailed below"
   }
   }

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::my-bucket"
            ]
        },
        {
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListMultipartUploadParts",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::my-bucket/*"
            ]
        },
        {
            "Action": [
                "kms:GetPublicKey",
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:GenerateDataKey",
                "kms:DescribeKey",
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:kms:eu-west-3:my-account-id:key/my bucket encryption key"
            ]
        }       
    ]
}

In addition, as mentioned in the aws documentation, the provider is assuming a role allowing to create the Snapshot repository:

provider "opensearch" {
  alias             = "iam"
  aws_access_key    = "xxx"
  aws_region        = "eu-west-3"
  aws_secret_key    = "xxx"
  aws_token         = "xxx"
  healthcheck       = false
  sign_aws_requests = true
  sniff             = false
  url               = "https://myopensearch.myprivatedns"
}

The assumed role

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "es:ESHttpDelete",
                "es:ESHttpPost",
                "es:ESHttpPatch",
                "es:ESHttpPut"
            ],
            "Resource": [
                "arn:aws:es:eu-west-3:123456789:domain/myopensearchdomain/*",
            ]
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::123456789:role/my-snapshot-role"
        }
    ]
}

What is the expected behavior?

No error thrown.

What is your host/environment?

AWS VPC

[anselsevier]

hi @fmartinou I too get the same error. I have reported in the OpenSearch forum . Also asked for an advice https://forum.opensearch.org/t/repository-exception-repository-type-s3-does-not-exist/17754/4

[anselsevier]

If you have any pointer to resolve this please let me know