sensu / uchiwa

Uchiwa is a simple yet effective open-source dashboard for the Sensu monitoring framework.
https://uchiwa.io
MIT License
920 stars 174 forks source link

Lack of health checking leads to loss of situational awareness #811

Closed cwjohnston closed 5 years ago

cwjohnston commented 5 years ago

Expected Behavior

Uchiwa uses Sensu /health API endpoint to determine the health of API hosts. If an API host is not healthy, Uchiwa doesn't attempt to poll it when refreshing a given datacenter.

Current Behavior

Unhealthy API hosts have an equal probability of being used on each polling interval.

Possible Solution

Implement health check with state tracking sufficient to avoid polling API hosts we know to be unhealthy.

Steps to Reproduce (for bugs)

  1. Configure a Sensu data center with two API hosts
  2. Disable the API on one of the hosts.
  3. Observe that you have a 50/50 chance of polling the unhealthy API host

Context

When Uchiwa fails to refresh data from a configured API host, visibility into the last known state is lost, making it even harder to respond to the kinds of error states Sensu is meant to monitor for.

Your Environment