search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.54k stars 1.75k forks source link

General questions about long term compatibility mainly in regards to ELK tools #879

Open tgurr opened 3 years ago

tgurr commented 3 years ago

Describe the question

https://github.com/opensearch-project/OpenSearch/pull/847 has been merged to retain support with external tools like e.g. Beats, Curator, Logstash, and so on. Various tools will probably be happy to merge code to also support OpenSearch, but are there plans for long term compatibility mainly with the tools provided/developed by elastic? For now the workaround will be to stay on Beats/Logstash 7.10.x but what happens if there'll be security issues, breakage, request for new features, afaiks support for the ELK stack 7.10.x ends on 11 May 2022. And the workaround in OpenSearch already has the comment will be removed in a future version. elastic upstream appears to be uncooperative as can be seen in PRs being closed that would make sure the tools will work with OpenSearch as well, e.g.: https://github.com/elastic/beats/pull/26305

Will OpenSearch/Amazon also jump in and maybe fork the remaining tools of the "ELK" stack or is this out of OpenSearchs interests? Will we end up running into unresolvable issues when building something with OpenSearch as a base and ELK tools for feeding the data?

I know there are other tools which could also ship log data to Elastic/OpenSearch, e.g. Telegraf, but I'm not aware of something like the various Beats (Winlogbeat/Journalbeat/Filebeat/Auditbeat) which also ship and setup neat Dashboards and provide nearly out-of-box experiences, and of course Logstash.

elfisher commented 3 years ago

Hi @tgurr, #847 was indeed merged to help OpenSearch be backwards compatible with external tools like Beats, Logstash, etc. A lot of people are using Logstash and Beats, and so we are taking multiple measures to ensure those are still going to be viable for a while. In addition to #847, we are building a Logstash output plugin for OpenSearch and we’ve added documentation on which version of Logstash and Beats to use with OpenSearch.

With that being said, we did a survey and saw there's a lot of excitement around other ingestion agents that are super open (e.g. Fluentd, Fluent Bit and Open Telemetry). We are considering investing energy to make those great and add much-loved Logstash and Beats features to them. Would you be interested in adopting an alternative to Logstash and Beats? If so what would be the most important features to add?

Also I want to note that we’ll support and help out anybody who wants to build any tools that work with the Open Search ecosystem. If you are interested in helping with any specific tools, let us know.

bgdnlp commented 3 years ago

On FreeBSD compatibility is a problem.

beats7 has been upgraded to 7.13 in "latest" pkg repository (think test) and will soon arrive in "quarterly" (think stable). At that point beats will fail to connect to any other ES distribution except Elastic's with no warning to users except for errors in log files. Not even the standard upgrade message about breaking changes. As far as I can tell there is no package for beats-oss.

fluent-bit is an option, but support for non-linux OSes seems to be a bit sketchy and it seems to lack some basic features, like adding month-year to index name when sending to ES.

I didn't test the others, but none of them are a drop-in replacement.

The features I'm using: