search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.65k stars 8.23k forks source link

Synthetic Monitoring: Multi-host Private Locations #185638

Open cedricremmicom opened 5 months ago

cedricremmicom commented 5 months ago

Multi-host Private Locations: We are looking into using Synthetic monitors a bit different than intended. Currently documentation specifies that synthetic monitors should be deployed against private locations which are linked to a fleet policy containing just 1 single host. We would like to divert from this specification and be able to deploy a private monitoring location to all the hosts in our fleet policy.

Describe a specific use case for the feature: We run an appliance at around 300 customer-sites. For example: we have docker on these appliances running a Solr container for various search operations. We would like to deploy a synthetic monitor to our fleet to monitor this service behind localhost:8983.

Currently we have already tried this out and in some way it does work, but you can clearly see in Kibana Alerts UI that this was not the intended configuration. For example: the alerts comming from these multi-host monitors are really confusing. The only field in the alert giving away on which host this alert was generated is the agent.name field, but this field isn't even available in the Alerts UI for filtering/grouping. The UI also doesn't allow to edit the internal synthetic monitor alerts. Without this restriction we would be able to fix this ourselves to support our desired configuration.

renzedj commented 5 months ago

I would like to see this as well, as it fits several of our use cases too. In the meantime, I've gotten around this by adding a logs-alias alias to the synthetics data streams. This allows me to use log rules for it, so I can group by agent.name or monitor.name. E.g.,

LOG VIEW Default
WHEN THE count
  OF LOG ENTRIES
WITH data_stream.dataset IS tcp
AND monitor.name IS my_monitor
AND state.status IS down

IS more than or equals 1
FOR THE LAST 5 minutes
GROUP BY agent.name

...and I set it to alert after two consecutive matches. I also set the test to alert disabled so that it doesn't trigger the Synthetics alert.

The bad thing is that the alert count doesn't show in the Synthetics view, and if one of the nodes is down, the whole test appears down.

elasticmachine commented 5 months ago

Pinging @elastic/unified-observability (Team:Observability)