search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
104 stars 80 forks source link

[BUG] Using beats to logstash, breaks the automatic parsing done by opensearch pipelines #213

Open RobinGoussey opened 1 year ago

RobinGoussey commented 1 year ago

Describe the bug When using logstash output plugin, and redirecting beats traffic to it, parsing does not happen anymore for filebeat modules. Filebeat seems to build pipelines in opensearch. When the output is opensearch. However, when using the logstash output, these pipelines are not used.

Eg, when using postgres module, and redirecting traffic to logstash, everything is in the message field, but it's not parsed.

To Reproduce Steps to reproduce the behavior: Set up a logstash, opensearch.

See the difference in letting the beat send to opensearch, in comparison with logstash.

Expected behavior Module logs should be parsed correctly.

Plugins None, except this one.

Additional context I don't know if this is a bug, or more of a feature request, but I see that in elasticsearch, there is a pipeline option in the output. So perhaps that can be added?

asifsmohammed commented 1 year ago

Hi @RobinGoussey, can you please share the config being used?