search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.13k stars 837 forks source link

Investigate elastic/opensearch forks #978

Open Jeffail opened 2 years ago

Jeffail commented 2 years ago

The dust is starting to settle now since the opensearch fork so it's probably worth spending some time understanding what's needed in order to continue to support both offerings. We're currently using https://github.com/olivere/elastic which has been fantastic (more info about future direction here: https://github.com/olivere/elastic/issues/1533) but we could also consider moving to the official client libraries as we only use a small fraction of ES endpoints.

Official libraries:

Questions:

gedw99 commented 2 years ago

Hey @Jeffail

Some people ( me too ) want to run their own Elastic equivalent.

There is a golang equivalent that matches the Elastic API closely https://github.com/prabhatsharma/zinc For collection they recommend https://github.com/fluent/fluent-logger-golang for logging into it

Its getting alot of attention. I guess tons of people want an efficient alternative.

It works well. It's based on bluge: https://github.com/blugelabs/bluge ( thankyou @mschoch ).

ryn9 commented 2 years ago

@Jeffail I would be interested in seeing a dedicated output for OpenSearch built against the aforementioned opensearch-go. We could ask the coordinators of https://opensearch.org/events/ to allow you to present, showing off Benthos and its OpenSearch integration.

arunx2 commented 2 years ago

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

mihaitodor commented 2 years ago

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

arunx2 commented 2 years ago

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

ryn9 commented 2 years ago

The official Go Client Library for OpenSearch - https://github.com/opensearch-project/opensearch-go

arunx2 commented 2 years ago

@Jeffail I can contribute in creating a dedicated opensearch output sink. Let me know.

Go ahead if you have cycles to spare. All new contributors are welcome! Happy to help you get it in shape for merging

I'm on it. Thanks!

@Jeffail here is the PR. https://github.com/benthosdev/benthos/pull/1388

mihaitodor commented 1 month ago

The opensearch output was released in v4.25.0.

We still need to migrate from olivere/elastic to https://github.com/elastic/go-elasticsearch.