search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
108 stars 4.93k forks source link

[Heartbeat] Implementing sharding for running monitors across multiple machines #40438

Open vigneshshanmugam opened 3 months ago

vigneshshanmugam commented 3 months ago

Proposal

There will be a single Heartbeat instance which will act as a coordinator that synchronously talks to remote endpoints that has the ability to run synthetics tests and write the results back.

Sharding will be implemented on the Heartbeat layer where given 10 monitors to run on the next scheduled run or across 100s of monitors, Heartbeat based on the number of machines behind a endpoint, Will shard and send the monitor configuration to the remote synthetic service endpoints using Web socket connection. The remote Synthetics runner will be responsible for running the synthetics monitors and writing the results via Web socket connection.

In addition to that, The connection must be encrypted between the Heartbeat coordinator and the Synthetics monitors runner, We should look in to allowing users to configure custom certificates if necessary to establish the connection.

elasticmachine commented 3 months ago

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)