Define Microservice Grafana Metrics Dashboard

osxhacker / demo-microsite

An exploration of microsite/microservice concepts.

BSD 3-Clause "New" or "Revised" License

2 stars 0 forks source link

Define Microservice Grafana Metrics Dashboard #8

Closed osxhacker closed 1 year ago

osxhacker commented 1 year ago

The system is collecting application metrics via Prometheus exposed by Kamon. Currently, there are dashboards which display "generic" metrics, such as for the OS within containers, but none reveal metrics specific to each microservice.

This enhancement will:

[x] Define at least one Grafana custom dashboard
[x] Have variables which select "all" or individual microservices
[x] Have one or more panels which show REST API metrics
[x] Have one or more panels which show Kafka metrics
[x] ~Have one or more panels which show scenario workflows~
[x] Have one or more panels which show HirakiCP metrics

osxhacker commented 1 year ago

The current HirakiCP configuration results in connection health checks to be performed once per minute. While this is a desired operation for detecting connection issues, what it results in is the metrics being skewed by those operations.

For example, the jdbc_pool_connections_borrowed_count over a 24 hour period of a largely idle storage-facility service is 1_301_601.

This significantly reduces the utility of monitoring HirakiCP metrics.

osxhacker commented 1 year ago

Displaying scenario-specific metrics is being deferred until additional AOP Advice are defined such that panels can be defined which are resilient to the absence of Prometheus measurements.