Closed cmacknz closed 1 year ago
Pinging @elastic/elastic-agent (Team:Elastic-Agent)
The ideal outcome here is that an system or integration test exists proving that a Filebeat instance running the shipper output can communicate with a shipper instance running the shipper input.
So, currently tinkering with filebeat to see how we can do this. There's a few open questions:
This is where the agent create's the shipper input configurations now: https://github.com/elastic/elastic-agent/blob/eaa9d0ac42986259e153482170e077ea8e191c5a/pkg/component/runtime/manager_shipper.go#L37-L51
Spoke today about how the design should work, I've updated the diagram in the description to match.
Summary of the implementation:
type
parameter in the input units it sends to the shipper to match the name of the shipper protocol input. This allows Filebeat to create and manage the lifecycle of the shipper inputs with no other modifications to the shipper. This is the code in the agent that needs to be modified.beat.Client
for each data stream. Each input receives a list of streams configured for a component along with the processors to apply to that stream. The input must be a map from data stream name to beat.Client to ensure the correct processors are applied to each event.Alternatives considered:
- The agent must set the
type
parameter in the input units it sends to the shipper to match the name of the shipper protocol input. This allows Filebeat to create and manage the lifecycle of the shipper inputs with no other modifications to the shipper. This is the code in the agent that needs to be modified.
This is actually fixed in this PR - https://github.com/elastic/elastic-agent/pull/2543
Great, then we just need to change the shipper spec file to have the name parameter match the input type we plan to run:
shipperSpecs[shipper.Name] = ShipperRuntimeSpec{
ShipperType: shipper.Name
shippers:
- name: shipper
description: "Elastic Agent Shipper"
platforms:
Create a prototype input for Filebeat that implements the shipper API. The intent is to implement the data shipper as another instance of Filebeat so that we can completely reuse the existing Beat event pipeline and outputs.
The shipper input will be Elastic licensed and should be added to the https://github.com/elastic/beats/tree/main/x-pack/filebeat/input directory.
The shipper input should ideally start a single gRPC server and create a new
beat.Client
for each unique data stream. This will allow each data stream to have its own processor configuration and allows for shared processor pipelines to execute concurrently. This follows the pattern used in the AWS S3 input in https://github.com/elastic/beats/pull/33658.We can explore other approaches if the ideal approach is initially too complex. The path of least resistance will likely be to have one instance of the shipper input for each input unit received from the Elastic Agent, where there is an input unit per component connected to the shipper. This will result in multiple gRPC server instances, but given that the shipper gRPC communication uses Unix domain sockets / named pipes this may be an acceptable overhead.
Acceptance Criteria