Open vnil1994 opened 2 years ago
I moved this to opensearch-clients.
I would also like to see an OpenSearch filter plugin implemented in Logstash. Is this feature being worked on yet?
We don't have anything in the works at the moment. Is anyone interested in picking up the work?
I've created a fork of the plugin that can talk to OpenSearch at https://github.com/jgough/logstash-filter-opensearch If someone wants to pick this up and pull it under the umbrella of the OpenSearch project I would be more than grateful.
You can download the gem from the releases there and it should hopefully work. Install it using logstash-plugin install logstash-filter-opensearch-0.1.0.gem
Then you can use it as you would expect:
filter {
opensearch {
hosts => ["http://opensearch:9200"]
index => "my-index"
sort => "id:asc"
query => "id: %{id}"
fields => {
"data" => "[my_index_field]"
}
}
}
Should now be able to install my plugin with the command logstash-plugin install logstash-filter-opensearch
@jgough Thanks! Would you be so kind to enable issues in your fork, and open (or I can open) an issue in your repo similar to https://github.com/opensearch-project/opensearch-plugin-template-java/issues/4 about moving that repo into the org?
Also we don't have to - if you're looking for more work to do, I would pickup a bunch of things from https://github.com/opensearch-project/.github to help grow a community around it (CONTRIBUTING, README, etc.).
@dblock I've enabled issues so please go ahead. Let me know what I can do to help with this
Hi,
This new opensearch
filter
plugin
for logstash
is still not working although installed successfully using bin/logstash-plugin install logstash-filter-opensearch
.
Can you please help?
Created a new topic in opensearch
community for the same- https://forum.opensearch.org/t/logstash-opensearch-filter-plugin/17756
Below are some relevant config and logs-
filter {
opensearch {
hosts => ["https://localhost:9200"]
#ssl => true
#ssl_certificate_verification => false
index => "students"
user => "admin"
password => "admin"
query_template => "es-query/student_id_query.json"
docinfo_fields => {
"_id" => "student_id"
}
}
}
Here the host is secured- https://localhost:9200
, ssl
config is disabled (as host directly contains https
) and ssl_certificate_verification
is disabled-
Error log:
[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Faraday::SSLError wrapped=#<OpenSSL::SSL::SSLError: certificate verify failed>>
Now, with the same config, if I enable ssl_certificate_verification
as ssl_certificate_verification => false
, i.e., the below,
filter {
opensearch {
hosts => ["https://localhost:9200"]
#ssl => true
ssl_certificate_verification => false
index => "students"
user => "admin"
password => "admin"
query_template => "es-query/student_id_query.json"
docinfo_fields => {
"_id" => "student_id"
}
}
}
then the below error is occurring-
Error log:
[ERROR][logstash.filters.opensearch] Unknown setting 'ssl_certificate_verification' for opensearch
FYI, I have used ssl => true
and host => ["localhost:9200"]
also, but got the same error.
Please note that the same settings works fine for ElasticSearch cluster v8.x
-
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html
Any suggestion will help me a lot.
Thanks and regards, Pushan.
@pushanDev I am afraid of not being of much help, if https://github.com/jgough/logstash-filter-opensearch is not working then you should open bugs there; looks like 7 months since last time a commit was made in that repo, so it will take someone motivated to pick it up
@pushanDev I am afraid of not being of much help, if https://github.com/jgough/logstash-filter-opensearch is not working then you should open bugs there; looks like 7 months since last time a commit was made in that repo, so it will take someone motivated to pick it up
Okay @dblock, thank you, I shall do that.
@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that logstash-output-opensearch
is maintained in the Opensearch Project but logstash-filter-opensearch
is not.
@dblock Still hoping someone official can pick this up and give it the attention it needs https://github.com/jgough/logstash-filter-opensearch/issues/1
@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that
logstash-output-opensearch
is maintained in the Opensearch Project butlogstash-filter-opensearch
is not.@dblock Still hoping someone official can pick this up and give it the attention it needs jgough/logstash-filter-opensearch#1
Thank you @jgough!
My last attempt at finding someone to pick it up and do work on it hasn't succeeded, so I don't think moving it into the org makes much sense. But if it does get maintainers outside we'll happily move it in.
@pushanDev Hi, I don't have much time myself (nor much ruby experience) to maintain the plugin and was hoping someone official from the project would step in to take over on this. It seems strange to me that
logstash-output-opensearch
is maintained in the Opensearch Project butlogstash-filter-opensearch
is not. @dblock Still hoping someone official can pick this up and give it the attention it needs jgough/logstash-filter-opensearch#1Thank you @jgough!
Hey, have you managed to solve the issue?
Is your feature request related to a problem? Please describe. We are running OpenSearch 1.2.2 and are looking to use the logstash filter plugin "elasticsearch" for lookup and enrichment in other indices, before we send the logs to Opensearch and it's corresponding index. However, it seems that the logstash filter plugin "Elasticsearch" does not support OpenSearch. It complains with the following error message:
I originally created a forum post here for this problem and was asked to create a feature request for it.
Describe the solution you'd like Attempt to let Logstash connect to an OpenSearch instance without failing the license check, or perhaps build a logstash plugin with features similar to the existing Elasticsearch filter plugin that is able to connect to Elasticsearch OSS and OpenSearch instances.
Describe alternatives you've considered We have tried running this command in the cluster:
PUT _cluster/settings
but the license check still fails.
Additional context The documentation for elasticsearch-filter-plugin: https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html