search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.49k stars 1.74k forks source link

ElasticSearch Filter Plugin #1976

Open vinaymaddi1269 opened 2 years ago

vinaymaddi1269 commented 2 years ago

Search Elasticsearch for a previous log event and copy some fields from it into the current event. Below are two complete examples of how this filter might be used.

Currently this filter is supported in Elastic eco system.

Use Cases and more details are present at https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

vinaymaddi1269 commented 2 years ago

I am looking for equivalent of ElasticSearch plugin for OpenSearch

RajivTaori commented 2 years ago

This should probably be filed for Data Prepper, since this is referring to a logstash filter plugin

dlvenable commented 2 years ago

This is different from Data Prepper. The Logstash Elasticsearch filter plugin will make a query to Elasticsearch and update the current event in Logstash with data from Elasticsearch.

I believe the ask here is to create and support an OpenSearch filter plugin for Logstash.

vinaymaddi1269 commented 2 years ago

This is different from Data Prepper. The Logstash Elasticsearch filter plugin will make a query to Elasticsearch and update the current event in Logstash with data from Elasticsearch.

I believe the ask here is to create and support an OpenSearch filter plugin for Logstash.

Thats correct

dlvenable commented 2 years ago

Thank you for the clarification. I created https://github.com/opensearch-project/data-prepper/issues/953 to see if something like this would be useful within Data Prepper.

anasalkouz commented 2 years ago

@dlvenable Shall we close this one on behalf of https://github.com/opensearch-project/data-prepper/issues/953?

dlvenable commented 2 years ago

@anasalkouz , No. This issue should stay open because the requestor is asking for a new Logstash plugin. Data Prepper is a distinct tool from Logstash.

anasalkouz commented 2 years ago

@elfisher can you please look into this and decide the next step?

elfisher commented 2 years ago

Is this the same as as the input plugin for logstash? https://github.com/opensearch-project/opensearch-plugins/issues/121

prashanttct07 commented 2 years ago

@elfisher yes, it is similar only difference is that belongs to input of logstash where as this requirement is for filter section of logstash.

elfisher commented 2 years ago

@prashanttct07 thanks for clarifying. I'll add this to the backlog, and we will work to prioritize it. If this is something you (or anyone else) would be interested in contributing, let us know. We are happy to help review.

gsinghk81 commented 2 years ago

Hi , Can we have the timelines for the availability of filter plugin

dblock commented 2 years ago

@gsinghk81 unless someone says otherwise nobody is currently working on this, so it doesn't have a timeline