ingest pipeline

[Alisher-Nabiev]

Hi, I am having trouble using an ingest pipeline. I can manually run the pipeline when reindexing the index, but I can't automatically ingest my index during creation.

this is my conf: output {

elasticsearch { host => 0.0.0.0 port => 9200 bulk_limit => 1000 bulk_timeout => 100 index_prefix => netflow pipeline => imei-pipeline data_type => netflow

testing my pipeline : POST /_ingest/pipeline/imei-pipeline/_simulate { "docs": [ { "_source": { "ipv4_src_addr": "10.1.1.1" } } ] }

res: { "docs" : [ { "doc" : { "_index" : "_index", "_type" : "_doc", "_id" : "_id", "_source" : { "ipv4_src_addr" : "10.1.1.1", "customer" : { "ipv4_src_addr" : "10.1.1.1", "IMEI" : "89898989898989", "CustomerName" : "some_brend", "ThingId" : "ThingId_ICCID_89898989898989" } }, "_ingest" : { "timestamp" : "2024-08-05T12:10:28.478468965Z" } } } ] }

when using reindex command: POST /_reindex?wait_for_completion=false { "source": { "index": "netflow-2024.08.02" }, "dest": { "index": "enriched_netflow-2024.08.02", "pipeline": "imei-pipeline" } }

---

so my main goal is to ingest the netflow index to be created from the beginning with the data from the pipeline (IMEI, customer,CustomerNam, ThingId) and avoid reindexing.

[lmangani]

I'm afraid we can't help with any such aspect but If you have a proposed solution, we might be able to implement it.

[Alisher-Nabiev]

this is the same as in logstash : https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-pipeline

[lmangani]

I don't know what you mean or require here. I guess this is not about pastash but rather about Elasticsearch since we don't do anything about the request body and/or its fields.