search

elastic / ingest-docs

Elastic Ingest Documentation
Other
4 stars 56 forks source link

[REQUEST]: Addition of connection handling details (overview) for Elastic Agent. #844

Open e-parth-pathak opened 7 months ago

e-parth-pathak commented 7 months ago

Description

Customer has opened up a support case requesting for some information on how Elastic Agent works and how do we handle connection loss and subsequent data losses. In his words:

  1. What are the interactions/requests Elastic Agent triggers against Fleet Server?

  2. How Elastic Agent handle offline/disconnect periods?

  3. How the components running Elastic Agent handle offline/disconnect periods? Do we lose data?

  4. For the log collection (filebeat) the connection be lost for hours, and then when connected again, the log data is sent from last successful checkpoint, so there is no data loss?

    • I understand we have filebeat registry in this context, would it work in the same way for Elastic Agent?

  5. For metrics, how would this work? I assume there would be some metric caching, hence there would be no data loss if connection lost for a short period of time, but there would be data loss for longer outages.

You can reach out to @lucabelluccini for further clarifications.

Resources

We have internal working very nicely documented for Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html

We have something for metricbeat as well: https://www.elastic.co/guide/en/beats/metricbeat/current/how-metricbeat-works.html

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @kilfoyle

kilfoyle commented 7 months ago

Thanks for opening this @e-parth-pathak, and for the helpful links! We'll definitely need input from the development team on this. For now I've added it to the platform docs backlog.