Support Amazon Opensearch Serverless Sink

[dinujoh]

Is your feature request related to a problem? Please describe. It would be nice to have support for Amazon Opensearch Serverless Sink. Amazon Opensearch supports preview for Serverless. As a developer, you can use OpenSearch Serverless to run petabyte-scale workloads without configuring, managing, and scaling OpenSearch clusters.

Describe the solution you'd like New plugin to support Amazon Opensearch Serverless as Sink.

Tasks

• [x] #1881
• [ ] #2281

[dlvenable]

I'm not sure we want a new sink here. We could make this a configuration on the existing opensearch sink.

One possible approach could be based on the solution for #877. In addition to opensearch and opendistro, Data Prepper could have an aws_serverless option here.

Or maybe this should be a completely different parameter entirely. Either way, I think we can make this work in the existing sink plugin.

Is there a particular reason you propose a different sink plugin?

[dlvenable]

In order to complete this, Data Prepper will need to support the x-amz-content-sha256 header. We could do this ourselves, but we want to update to use the AwsSdk2Transport from the opensearch-java client anyway.

The opensearch-java client does support the x-amz-content-sha256 header as provided here: https://github.com/opensearch-project/opensearch-java/pull/339 and this was included in the 2.2.0 release.

Additionally, Serverless uses the aoss service name instead of es. The AwsSdk2Transport also supports setting the service name via the signingServiceName property:

https://github.com/opensearch-project/opensearch-java/blob/00d8f993862f2385ca26dd9baaab4eebeb721932/java-client/src/main/java/org/opensearch/client/transport/aws/AwsSdk2Transport.java#L160-L165

In order to support Serverless, Data Prepper needs to:

1. Update to opensearch-java 2.2.0
2. Refactor the OpenSearch sink to use the AwsSdk2Transport - See: #1881
3. Provide a service_name property which defaults to es, but can be changed by pipeline authors.