"authentication/authorization failure" while connecting to OpenSearch Serverless using IAM role

[simpledev123]

Steps to reproduce:

Attach an IAM role with all permissions.

**aoss:CreateCollectionItems aoss:DeleteCollectionItems aoss:UpdateCollectionItems aoss:DescribeCollectionItems

aoss:CreateIndex aoss:DeleteIndex aoss:UpdateIndex aoss:DescribeIndex aoss:ReadDocument aoss:WriteDocument**

Run below Java class within a container.

import java.io.IOException;
import java.util.Optional;

import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.OpenSearchException;
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
import org.opensearch.client.transport.aws.AwsSdk2Transport;
import org.opensearch.client.transport.aws.AwsSdk2TransportOptions;

import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsSessionCredentials;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.auth.signer.params.Aws4SignerParams;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.regions.Region;

public class IndexDocument {

    private static final String host = "<test-domain>.us-west-2.aoss.amazonaws.com";
    private static Region region = Region.US_WEST_2;

    public static void main(String[] args) {
        SdkHttpClient httpClient = ApacheHttpClient.builder().build();
        try {

        //testing out the default credential chain
            final AwsCredentialsProvider credentials = Optional.ofNullable(AwsSdk2TransportOptions.builder().build())
                    .map(o -> o instanceof AwsSdk2TransportOptions ? ((AwsSdk2TransportOptions) o) : null)
                    .map(AwsSdk2TransportOptions::credentials)
                    .or(() -> Optional.ofNullable(AwsSdk2TransportOptions.builder().build().credentials()))
                    .orElse(DefaultCredentialsProvider.create());

            Aws4SignerParams signerParams = Aws4SignerParams.builder()
                    .awsCredentials(credentials.resolveCredentials())
                    .signingName("aoss")
                    .signingRegion(region)
                    .build();
           //pring cred values
            AwsSessionCredentials cred = (AwsSessionCredentials) signerParams.awsCredentials();
            System.out.println(cred.);
            System.out.println(cred.secretAccessKey());
            System.out.println(cred.sessionToken());

            OpenSearchClient client = new OpenSearchClient(
                    new AwsSdk2Transport(
                            httpClient,
                            host,
                            "aoss",
                            region,
                            AwsSdk2TransportOptions.builder().build()));

            try {
                try {
                    client.cat().indices();
                } catch (IOException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            } catch (OpenSearchException e) {
                // TODO Auto-generated catch block
                System.out.println("ERROR" + e.error().reason());
            }

        } finally {
            httpClient.close();
        }
    }
}

AWS supplies session access & secret ids along with session token. That will get printed. But final result would be a 403 error with message "authentication/authorization failure".

This not a permission issue in IAM role. If we pass access key & secret key (generate with same permissions) in environment variable, the client works fine.

Also Python opensearch-py library works just fine within the same container with session credentials from same IAM role.

Python code:

from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3 
client = boto3.client('opensearchserverless')
service = 'aoss'
region = "us-west-2"
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,region, service, session_token=credentials.token)
client = OpenSearch(hosts=[{'host': "<test>.us-west-2.aoss.amazonaws.com", "port": 443}],http_auth=awsauth,use_ssl=True,verify_certs=True, connection_class=RequestsHttpConnection,timeout=300)
client.indices.get_mapping()

Have anyone faced this issue while accessing OpenSearch serverless with opensearch-java 2.2.0? Is there any workaround or planned fix?

[dblock]

I have the exact same code working in https://github.com/dblock/opensearch-java-client-demo and was able to cat indices (I do have different IAM role/permissions, but you already verified that it works with other clients). Let's see if this is something in the setup of your application? Try running my demo without changes, does it work or do you get the same error?

[simpledev123]

I have tried the same code. Only change was that I didn't pass the credentials in environment variable. It was picked up by default credential chain from IAM role attached. It didn't work.

[dblock]

@simpledev123 Have you tried the demo or the same code? Check out, and try running running the demo? Instructions in README. Since the code is identical I suspect some other problem.