search

elastic / package-spec

EPR package specifications
Other
18 stars 73 forks source link

[Change Proposal] Add support for ingest pipelines in transforms #833

Open jsoriano opened 1 week ago

jsoriano commented 1 week ago

Transforms allow to define ingest pipelines, so documents are processed before ingestion. This allows for example to remove fields that are not relevant in the transform, or to adjust fields intended to have different values, as in https://github.com/elastic/elastic-package/issues/2218#issuecomment-2477128353.

This is partly supported now, a destination index can have a pipeline, but currently it needs to be hard-coded:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: "1.23.0-latest_ioc"

We should allow to use templates there as we allow in other places, so configuration can be something like this:

dest:
  index: "logs-ti_anomali_latest.threatstream-3"
  aliases:
    - alias: "logs-ti_anomali_latest.threatstream"
      move_on_creation: true
  pipeline: '{{ IngestPipeline "latest_ioc" }}'

This may not need changes in the spec but will need changes in Fleet.