search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.08k stars 4.89k forks source link

Migrate filebeat and metricbeat to the V2 agent specification file format #31900

Open cmacknz opened 2 years ago

cmacknz commented 2 years ago

As part of the V2 architecture changes the agent specification file format is being redesigned to remove automatic configuration transformations like inject_stream_processor.

Both filebeat and metricbeat currently rely on a variety of configuration file transformations when run under agent which must be removed:

The scope of this issue is to move the logic for each of these configuration transformations into the beat itself when it detects it is being run under agent (for example, seeing the -management.enabled flag on the command line).

Relates To:

This task is considered complete when a test exists proving that filebeat and metricbeat behave correctly when started using their V2 specification file. This includes but is not limited to testing that:

elasticmachine commented 2 years ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

cmacknz commented 2 years ago

Assigning to @rdner as I think the work here will probably be a natural continuation of https://github.com/elastic/beats/issues/33135 for the shipper.

cmacknz commented 2 years ago

There is a chance this overlaps with the work needed for https://github.com/elastic/beats/issues/31901, ideally they can be done independently but we'll need to sync up first.

cmacknz commented 2 years ago

I think it will make sense for @fearful-symmetry to take this over as it is linked strongly to #31901

fearful-symmetry commented 2 years ago

Yah, still trying to figure out how to grapple with this.

@cmacknz I assume the goal here is to treat the V2 config system as the "default" for how things work, meaning we make the beats backwards-compatible, and future V2 imputs need no config transformation to start with.

I'm undecided on if we can/should "port" that transformation code/spec config into the beats, and have them basically apply the transformations once we get the config from agent. The alternative would be something more piecemeal with custom code for each beat.

cmacknz commented 2 years ago

@cmacknz I assume the goal here is to treat the V2 config system as the "default" for how things work, meaning we make the beats backwards-compatible, and future V2 imputs need no config transformation to start with.

👍 Yes, we want to obsolete the V1 spec files and only use the V2 specs that do not allow transformations.