The agent currently configures a set of default processors to apply globally to all events collected from each Beat. See https://github.com/elastic/beats/pull/34149 for the original implementation.
Several of these processors are expensive to create and will cache copies of the same metadata in each Beat they run in. See https://github.com/elastic/beats/issues/35000 for an example of a performance issue introduced by creating too many instances of these processors.
We should migrate these processors into the shipper to minimize the number of instances of them that need to be run, and allow them to apply to all data collected by agent.
One challenge will be that the default processor configuration for each beat is almost but not quite the same. We will need to evaluate whether these differences are significant enough to warrant apply the processors for all data routed through the shipper or only for data routed through the shipper from a specific Beat.
The agent currently configures a set of default processors to apply globally to all events collected from each Beat. See https://github.com/elastic/beats/pull/34149 for the original implementation.
Several of these processors are expensive to create and will cache copies of the same metadata in each Beat they run in. See https://github.com/elastic/beats/issues/35000 for an example of a performance issue introduced by creating too many instances of these processors.
We should migrate these processors into the shipper to minimize the number of instances of them that need to be run, and allow them to apply to all data collected by agent.
One challenge will be that the default processor configuration for each beat is almost but not quite the same. We will need to evaluate whether these differences are significant enough to warrant apply the processors for all data routed through the shipper or only for data routed through the shipper from a specific Beat.