Dependency health checking

ebensom commented 9 months ago

Description

Implement periodic health checking of compass dependency by periodically querying the oauth2 protected compass-director graphql API endpoint in a separate goroutine and keep the latest check result up-to-date. Expose the current (up-to-date) healthcheck result on the Prometheus metrics endpoint via series like:

{app}_{subsys}_compass_director_health{url="..", status="healthy"} 1
{app}_{subsys}_compass_director_health{url="..", status="error"} 0
{app}_{subsys}_compass_director_health{url="..", status="unknown"} 0

Reasons

Ability to cross-correlate compass-manager errors with director (dependency) errors.

Attachments

tobiscr commented 7 months ago

@ebensom : Please monitor the compass director API directly to detect service outages.

We will on our side implement following new metrics:

[ ] Measuring failure rate for API calls to compass director
[ ] Configure alert if requests to compass directory fail for 5 times in a row (we trigger the alert configuration SRE side and support them)

Disper commented 6 months ago

kyma-project / compass-manager

Dependency health checking #112