Opensearch support

[arnitolog]

I tried to configure kafka-connect-elasticsearch 11.1.2 with my cluster which is built with OpenSearch binaries and got an error:

Connector version 11.1.2 is not compatible with Elasticsearch version 1.0.0. Elasticsearch version must be at least 7.0.0.

Are there any plans to support OpenSearch: https://opensearch.org ?

[kpatelatwork]

Thanks, @arnitolog for the proposal. At this point, it's not in our roadmap but we can consider it in the future.

[LeaderXFX]

I second the request. AWS has already implemented OpenSearch.

[christopher-johnson]

AWS OpenSearch supports only Elasticsearch clients up to 7.13.4 in compatibility mode: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/samplecode.html#client-compatibility

[dmathieu]

I'll second this request. A lot of providers are moving from ES to OS, and this is going to be a blocker on all of those. However, I don't think this plugin should support OpenSearch, as the APIs will probably start diverging in the future. Instead, there should be a fork of this library using the AWS libraries, and supporting OS only.

[dmathieu]

Hi!

We were in a dire need of this, and since no proper solution exists today, we have decided to fork this repository and bring support for OpenSearch on it. You can find the fork here: https://github.com/dmathieu/kafka-connect-opensearch

We're happy to maintain the fork. We'd also be happy to have Confluent take ownership.

[kuldeeprishi]

+1 for opensearch compatibility

[jacojoubert12]

The link directs to a deprecated repo and the suggested one in that README does not work.

Failed to find any class that implements Connector and which name matches io.aiven.kafka.connect.opensearch.OpensearchSinkConnector

[noudAndi]

+1 for OpenSearch support.

Right now, there is no way to connect to AWS OpenSearch neither to the "old" 7.10 version nor to any OpenSearch Versions.

Using Elasticsearch 7.1:

FAILED
Unable to validate configuration. If an update was made to the configuration, this means that the configuration was invalid, and the connector continues to operate on a previous configuration that passed validation. Errors:
connection.url: Could not connect to Elasticsearch. Error message: Invalid or missing build flavor [oss]
connection.url: Failed to create client to verify connection. Invalid or missing build flavor [oss]
connection.username: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: Invalid or missing build flavor [oss]
connection.password: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: Invalid or missing build flavor [oss]

Using OpenSearch 1.3:

FAILED
Unable to validate configuration. If an update was made to the configuration, this means that the configuration was invalid, and the connector continues to operate on a previous configuration that passed validation. Errors:
connection.url: Could not connect to Elasticsearch. Error message: Invalid or missing tagline [The OpenSearch Project: https://opensearch.org/]
connection.url: Failed to create client to verify connection. Invalid or missing tagline [The OpenSearch Project: https://opensearch.org/]
connection.username: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: Invalid or missing tagline [The OpenSearch Project: https://opensearch.org/]
connection.password: Could not authenticate the user. Check the 'connection.username' and 'connection.password'. Error message: Invalid or missing tagline [The OpenSearch Project: https://opensearch.org/]

[passuied]

+1 for OpenSearch Support

[Rayshard]

+1 for OpenSearch support

[wdxxl]

+1 for OpenSearch support

[msardara]

+1 for OpenSearch support