search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.17k stars 4.92k forks source link

Inject Beat gRPC stream identifiers without relying on agent specification file transformations #33135

Closed cmacknz closed 2 years ago

cmacknz commented 2 years ago

To quickly allow metricbeat and filebeat to integrate with the shipper the stream identifiers (data stream and input ID) needed to allow the shipper to map events to processors are being injected via the inject_stream_processor specification file transformation. See the implementation in https://github.com/elastic/elastic-agent/pull/527.

In the V2 agent specification file format the agent will no longer be able automatically transform sections of the agent policy into a beat configuration file. All inputs must know how to configure themselves to run under agent automatically.

Define and implement a way of providing the needed stream identifiers to agent inputs without relying on specification file transformations as required by the V2 specification file format. This includes all beats (not just filebeat and metricbeat) and non-beat agent inputs like endpoint security.

Ideally the beats can consume the agent policy sections containing the relevant information directly without transformation, and instantiate the necessary processors automatically. This is similar to the approach heartbeat currently uses, see the heartbeat specification file and heartbeat code adding the processors.

Relates To:

This task is considered complete when:

  1. A test exists proving that the shipper can map processors to events sent from filebeat and metricbeat without using any configuration file transformations.
  2. The implementation used for metricbeat and filebeat is available for reuse in libbeat or elastic-agent-libs, such that any existing beat can easily be updated to add the new stream identifiers to produced events when communicating with the shipper.
cmacknz commented 2 years ago

@fearful-symmetry I believe the work remaining here is to use a config transformation to automatically add the metadata add_fields processors from https://github.com/elastic/elastic-agent/pull/527 to each of the Beats on the v2 feature branch.

fearful-symmetry commented 2 years ago

@cmacknz A little confused here. The processors should all be added on the beats side, are you talking about importing them into the shipper?

cmacknz commented 2 years ago

This issue was created before we had any of the v2 implementation in Beats. You are correct that the remaining work should happen in Beats, this issue will be transferred to the Beats repository.

cmacknz commented 2 years ago

Synced up with Alex, this has already been taken care of. Closing.

https://github.com/elastic/beats/blob/a528bd212543af278528e0130ed3b34d7668527f/x-pack/libbeat/management/generate.go#L173-L176

https://github.com/elastic/beats/blob/a528bd212543af278528e0130ed3b34d7668527f/x-pack/libbeat/management/generate.go#L194-L195