[Meta] Elastic Agent Shipper Project - Githubissues

elastic / elastic-agent-shipper

Data shipper for the Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.

Other

9 stars 17 forks source link

[Meta] Elastic Agent Shipper Project #16

Open cmacknz opened 2 years ago

cmacknz commented 2 years ago

Experimental, Beta, and GA Criteria

[ ] https://github.com/elastic/elastic-agent-shipper/issues/197

Blockers for Beta Milestone 1 - Elasticsearch output and memory queue

NEW - Convert the shipper to a Filebeat input

[x] https://github.com/elastic/beats/issues/35135
[ ] https://github.com/elastic/elastic-agent/issues/2521
[ ] https://github.com/elastic/beats/issues/35615 (enables setting shipper queue config via agent policy)
[ ] https://github.com/elastic/elastic-agent/issues/2581
[ ] https://github.com/elastic/elastic-agent/issues/2654
[ ] https://github.com/elastic/elastic-agent/issues/2580
[ ] https://github.com/elastic/beats/issues/35266
[ ] https://github.com/elastic/beats/issues/35267
[x] https://github.com/elastic/elastic-agent-shipper/issues/306
[ ] https://github.com/elastic/beats/issues/35615

Bugs

Features

Elasticsearch Output V2

Monitoring

Performance

Testing

Technical Debt

Documentation

[ ] Document the shipper process and how it interacts with agent policy:
[ ] Document the shipper feature flag and the current stability guarantees.
[ ] Document why a user would want to enable the shipper.
[ ] Document how the number of shippers (and therefore output connections) can be controlled via the agent policy.
[ ] Document configuration of the shipper queues.
[ ] Document any backwards compatibility or migration issues when moving from per beat outputs and queues to the centralized shipper.

Future Feature Milestones

Kafka Output

[x] https://github.com/elastic/kibana/issues/143324
[ ] https://github.com/elastic/elastic-agent-shipper/issues/168
[ ] https://github.com/elastic/elastic-agent-shipper/issues/232
[ ] https://github.com/elastic/elastic-agent-shipper/issues/203
[ ] Add support for metrics observers, add output metrics: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/kafka/client.go#L41
[ ] Add support for the retrying failed events: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/kafka/client.go#L48-L49
[ ] Support event codecs: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/kafka/client.go#L63
[ ] Support partition caching: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/kafka/partition.go#L122-L125
[ ] Update sarama to the latest version
[ ] Compatibility testing between existing Beats Kafka output configurations and the shipper Kafka output implementation.

Logstash Output

[x] https://github.com/elastic/elastic-agent-shipper/issues/129
[ ] Add support for configuring the retry algorithm: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/config.go#L29
[ ] Support configuring bulk max size: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/config.go#L21
[ ] Support load balancing between hosts: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/config.go#L20
[ ] Support pipeline with the async client: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/config.go#L25
[ ] Support slow start mode: https://github.com/elastic/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/config.go#L22
[ ] Add event codec support: https://github.com/cmacknz/elastic-agent-shipper/blob/de8cd7465741ff3a3b81476994f2155cc674850e/output/logstash/encode.go#L11-L12
[ ] Compatibility testing between existing Logstash output configurations and the shipper Logstash output implementation.
[ ] Performance test the Logstash output.

Global Processors

[x] https://github.com/elastic/elastic-agent-shipper/issues/161
[ ] https://github.com/elastic/elastic-agent/issues/1688
[ ] https://github.com/elastic/elastic-agent-shipper/issues/292
[ ] https://github.com/elastic/elastic-agent/issues/1689
[ ] https://github.com/elastic/elastic-agent/issues/1690
[ ] Allow configuring global processors that apply to all events: https://github.com/elastic/ingest-dev/issues/2442.

Endpoint Security Support

[ ] https://github.com/elastic/elastic-agent-shipper/issues/213
[x] https://github.com/elastic/elastic-agent-shipper/issues/275
[ ] Require use of an encrypted disk queue when Elastic Defend is installed

Disk Queue

[ ] https://github.com/elastic/elastic-agent/issues/1656
[ ] elastic/elastic-agent-shipper#122
[x] elastic/elastic-agent-shipper#27
[ ] Immediately acknowledge events persisted to the disk queue when an input is subscribed to the acknowledgement stream.
[x] https://github.com/elastic/elastic-agent-shipper/issues/27
[x] https://github.com/elastic/elastic-agent-shipper/issues/49
[ ] https://github.com/elastic/beats/issues/30069
[ ] https://github.com/elastic/elastic-agent-shipper/issues/125
[ ] https://github.com/elastic/elastic-agent-shipper/issues/145

Future Enhancements - Not Yet Planned

softengchick commented 2 years ago

Brian McKinney and Michelle Bennett will be working this for Endpoint. @brian-mckinney @softengchick

cmacknz commented 2 years ago

@softengchick @brian-mckinney is there an issue for this on the endpoint side we can follow?

If you have not seen it recently, the gRPC API for the shipper is defined at https://github.com/elastic/elastic-agent-shipper-client/tree/main/api. The current API is meant to be generic but it is tailored to the needs of Beats since that is what we started with. It is possible we missed something that could make integrating with Endpoint easier.

Reach out once you are ready to start work with any questions you have. I expect this will be a larger change for Endpoint than the move to the V2 control protocol since it isn't just iterating on something Endpoint already supports.