Opensearch sink: support ingest pipeline configuration option

[sandervandegeijn]

Is your feature request related to a problem? Please describe. In opensearch you can use ingest pipelines to process data. Fluent-bit, logstash etc support this via a property on the output/sink configuration. Data prepper does not.

Describe the solution you'd like ingest_pipeline option for the opensearch sink

Describe alternatives you've considered (Optional) Index templates through dataprepper, but this is overkill imho.

Additional context N/A

[dlvenable]

To be sure, you mean using this feature, correct? https://opensearch.org/docs/latest/ingest-pipelines/index/

Also, is there a particular reason you want to use an ingest-pipeline along with Data Prepper? I would generally recommend using Data Prepper instead of ingest-pipelines.

[sandervandegeijn]

Hi, yeah that's the one. Sometimes dataprepper is the choice, but not always. Ingest pipelines an supported and actively developed feature in opensearch which have their uses. i.e. the Elastic log source use ingest pipelines for data manipulation, so we have loaded them into opensearch. Every other log forwarder supports it (logstash plugin, fluent-bit, fluent-d, etc)

This is holding me back from stripping logstash-oss from our egress path from Kafka. Implementation can't be that hard I hope?