User Facing Metrics Service

[davidffrench]

Context

Red Hats managed services should have a consistent user experience for exposing customers service instance metrics. This should be provided in both a JSON and Prometheus text format. Creating a new user-facing metrics service, exposing a subset of managed service instance metrics to the customers will allow the customers to monitor their own managed service instances. It will also allow UIs to display their instance metrics in user-friendly dashboards.

User Narrative

Peter is the team lead for Innovation Inc, which has purchased several Red Hat managed services including Red Hat OpenShift Streams for Apache Kafka (RHOSAK). Peter is using RHOSAK as part of their own internal system and would like to monitor the Kafka instance and include the metrics in their own internal system dashboards. Thankfully, Peter can easily do this by scraping their Kafka instance metrics from the Red Hat user-facing metrics API directly into their companies Prometheus instance.

Steve from Innovation Inc is also interested in viewing his RHOSAK instance metrics, he is happy to view these from his Kafka instance page available on console.redhat.com where he can see several widgets displaying his instances metrics. Curious about where these are coming from, he can see they are retrieved from the /query and query_range endpoints from the Red Hat user-facing metrics API.

Job Stories

• As a managed service developer, I want onboarding documentation for the user-facing metrics service, so that I know how I can expose a subset of my tenant metrics that are user-facing
• As a managed service developer, I want my user-facing metrics retrieved from Observatorium, so that I can send metrics to a single location
• As a managed service developer, I want the authorisation configuration documented, so that I can define which users or organisations are allowed to retrieve their metrics
• As an end-user, I want to retrieve metrics in a Prometheus text format, so that I can scrape my services metrics to my own Prometheus instance
• As an end-user, I want to retrieve metrics using a query or query_range endpoint, so that I can accurately read the metrics

Analysis

(links to analysis docs containing architecture design work, requirements gathering, etc)

Task List

• [ ]

[miguelsorianod]

Isn't this potentially covered by Observatorium??

[davidffrench]

Unfortunately, Observatorium does not currently authorise external users to their API. The idea is hopefully to have a conversation with the RHOBS team to extend Observatorium with this functionality.

[miguelsorianod]

Unfortunately, Observatorium does not currently authorise external users to their API. The idea is hopefully to have a conversation with the RHOBS team to build out this service.

I am not sure I understood this. Could you give a little bit more detail about it? At the moment we are able to write and read metrics to Observatorium. Additionally, I understand you are talking about the Red Hat hosted Observatorium instance. Observatorium is an open-source project and its api mentions writing metrics(https://github.com/observatorium/api). I think it is important we differentiate the open-source software vs limitations that are imposed on Red Hat-hosted versions of the software

[davidffrench]

I think it is important we differentiate the open-source software vs limitations that are imposed on Red Hat-hosted versions of the software

Yes, you are correct. This GitHub issue might be better off as an internal issue. @emmanuelbernard Do you have any suggestions on how to track these types of issues?

[emmanuelbernard]

The way to track it is to think slightly generalized so that it addressed the need a user (Red Hat here) has. A service will store lots of metrics and our de facto preference is the observatorium stack today (but any prometheus would do). As a service you want to expose a user centric subset of these metrics and make sure they are exposed via an API with the proper authz. How do you do it assuming your metrics system does only expose metrics internally?

[emmanuelbernard]

One thing that is striking me is that this job story only refers to documentation. Wouldn't it be some endpoint / proxying / authz pre flight checking that a fleet manager implementor would rather have done for him or her?

[davidffrench]

That is a fair assessment @emmanuelbernard .

I raised this functionality with the Observatorium team and shared this GitHub issue. The action is for the team to create a JIRA specifically for this which should elaborate on the job stories. Once that has been created, we should be able to close this specific issue.