search

elastic / elastic-agent-shipper

Data shipper for the Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
9 stars 19 forks source link

Migrate the default global Beat processors to the shipper #292

Open cmacknz opened 1 year ago

cmacknz commented 1 year ago

The agent currently configures a set of default processors to apply globally to all events collected from each Beat. See https://github.com/elastic/beats/pull/34149 for the original implementation.

Several of these processors are expensive to create and will cache copies of the same metadata in each Beat they run in. See https://github.com/elastic/beats/issues/35000 for an example of a performance issue introduced by creating too many instances of these processors.

We should migrate these processors into the shipper to minimize the number of instances of them that need to be run, and allow them to apply to all data collected by agent.

One challenge will be that the default processor configuration for each beat is almost but not quite the same. We will need to evaluate whether these differences are significant enough to warrant apply the processors for all data routed through the shipper or only for data routed through the shipper from a specific Beat.

cmacknz commented 1 year ago

Copying my comment from https://github.com/elastic/beats/pull/35318/files#r1186277241

The complexity to this comes from:

  1. If someone disables the shipper the default processors need to go back in the inputs
  2. The default processor configurations vary somewhat by Beat
  3. The shipper's default processors are true global processors and will apply to events the did not before, like endpoint security.