search

opensearch-project / documentation-website

The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.
https://opensearch.org/docs
Apache License 2.0
68 stars 473 forks source link

Update docs with more cautions about beats compatability #119

Closed ryn9 closed 3 years ago

ryn9 commented 3 years ago

Please update the docs to reflect beyond wire compatibility, using beats beyond 7.10.x risks beats using logic that may not be compatible with the opensearch backend.

Example - beats often make use of ingest pipelines - and newer ingest pipelines may use features that require the opensearch backend be compatible with > 7.10.x

elfisher commented 3 years ago

@ryn9 is the logic you are referring to in the open source beats?

ryn9 commented 3 years ago

@elfisher - it is applicable to both opensource and non-opensource versions

aetter commented 3 years ago

Hi @ryn9, I added a note to this page: https://opensearch.org/docs/clients/agents-and-ingestion-tools/index/#downloads

Is your thinking here that we should just recommend 7.10.2 across the board?

ryn9 commented 3 years ago

I would recommend at least a caution that use of beats > 7.10 risks potential compatibility issues with use of ingest pipelines. Elastic has pushed a fair amount of ETL into the ingest pipelines that lives inside of Elasticsearch, instead of ETL being done in beats, logstash, or other part of the data ingest process. As such - there very well could be ingest pipeline code not compatible with Opensearch. Again - this is applicable to both opensource and non-opensource versions of beats. Hopefully I am being clear - but do let me know if you need more info.