search

elastic / package-spec

EPR package specifications
Other
17 stars 72 forks source link

Support for shared fields mapping and pipelines between data streams #89

Open narph opened 3 years ago

narph commented 3 years ago

We have several cases where we need to define mapping for fields at package level, atm we have to define them under each data stream, same goes for ecs fields like message or error.message etc. Would be great to provide a mapping at package level or even higher when it comes to ecs fields.

Same situation for pipelines, in some cases same processing of fields is required in all data streams for an integration, atm we have to define the shared pipeline at each data stream level.

ruflin commented 3 years ago

I would suggest to first solve this issue in integrations with automation, meaning during the build step these are copied / duplicated to the right place.

We should also discuss if we need it in package-spec and with it support it in Kibana but it will make things a bit more complicated.

ycombinator commented 3 years ago

Yeah, I think we could generalize this notion to allow references to other files, like includes, from any package file. Then as part of elastic-package build we could resolve these includes by inlining the referenced content.