Istio and MicroProfile Metrics

[mikecroft]

Metrics provides Prometheus format metrics. Istio provides an optional Prometheus instance for metrics.

Does Metrics need to provide anything so that Prometheus in a k8s/Service Mesh environment can play nicely? Metrics currently publishes metrics to the endpoint that the default Prometheus scraper already expects, so outside of a service mesh environment, the configuration is very simple. It may be that this is already optimal.

[pilhuhn]

One thing that comes to mind could be additional labels

[mikecroft]

Labels is an interesting idea; we could add labels which reflect details about the host or container where the runtime is located. Perhaps details like k8s pods/services as well, if they are present, so metrics can be correlated easily per service in something like Grafana.

Thinking of Google's SRE four golden signals, having a grouping of a k8s service would be very helpful for creating good dashboards, IMO.

[mikecroft]

paging @brian-brazil - as someone with a lot of Prometheus experience, we'd love to hear your thoughts on this if you have any!

[brian-brazil]

Labels is an interesting idea; we could add labels which reflect details about the host or container where the runtime is located. Perhaps details like k8s pods/services as well, if they are present, so metrics can be correlated easily per service in something like Grafana.

That's the role of target labels on the Prometheus side, an application's /metrics shouldn't expose these.

[Emily-Jiang]

We can add @Counted(monotonic=true) on the JAX-RS endpoint of service A. should do the similar thing for service b.

@GET
    @Retry
    @Fallback(fallbackMethod="serviceAFallback")
    @Produces(MediaType.TEXT_PLAIN)
public String callServiceB() throws Exception {

[kenfinnigan]

I think the main issue to investigate is whether /metrics can easily be pushed to Istio Prometheus.

I believe Istio Metrics requires special config to expose each individual metric