Prometheus Receiver UI for Viewing Targets/Service Discovery/Config

[gracewehner]

Currently, authoring scrape configs or debugging issues with scrape configs can be difficult and has a learning curve using just the debug logs of the prometheus receiver.

Prometheus has a web UI with the pages Targets, Service Discovery, and Configuration for this:

This UI uses html templates with the data filled by calling the scrape manager's functions. The prometheus receiver is using the same scrape manager in metrics_receiver.go, so the same functions to get the same data can be called to have an optional, similar UI.

A config option can be added for the prometheus receiver to enable hosting the UI as a local endpoint. The code that enables this would be in the Start() function of metrics_receiver.go, where the scrapeManager is created and run:

webOptions := web.Options{
    ScrapeManager: scrapeManager,
    Context: ctxWeb,
    ListenAddress: ":9090",
    ExternalURL: &url.URL{
        Scheme: "http",
        Host:   "localhost:9090",
        Path:   "",
    },
    RoutePrefix:    "/",
    ReadTimeout: time.Second * 30,
    PageTitle: "Prometheus Receiver",
    Version: &web.PrometheusVersion{
        Version:   version.Version,
        Revision:  version.Revision,
        Branch:    version.Branch,
        BuildUser: version.BuildUser,
        BuildDate: version.BuildDate,
        GoVersion: version.GoVersion,
    },
    MaxConnections: 5,
} 
webHandler := web.New(go_kit_logger, &webOptions)
listener, err := webHandler.Listener()
if err != nil {
    return errors.Wrapf(err, "error creating listener")
}
webHandler.ApplyConfig(r.cfg.PrometheusConfig)
webHandler.Ready()
go func() {
    if err := webHandler.Run(ctxWeb, listener, ""); err != nil {
        return errors.Wrapf(err, "error starting web server")
    }
}()

I currently have two different proof of concepts:

• Directly using the web package of the Prometheus repo, creating the webHandler, passing it the configuration options including the scrapeManager, and running it. The only code changes needed would be something similar to above.

  The issues with this approach are that:

  • The pages for Alerts, Graph, Rules, etc. won't apply in this situation and won't load any information.
  • The html templates and static files would need to be present in the working directory where the opentelemetry-collector executable is actually run because:
  • Prometheus generates a golang file that has the file system at build time: https://github.com/prometheus/prometheus/blob/main/Makefile#L53-L59 https://github.com/prometheus/prometheus/blob/main/web/ui/assets_generate.go. https://github.com/prometheus/prometheus/blob/main/web/web.go#L371. However, this file isn't available by using the UI package, so generating it ourselves would still require a change in the code since the web package will point to this Assets() function instead: https://github.com/prometheus/prometheus/blob/5a4c3734b10cccecdd17c8c4c3a43e0e1ee77466/web/ui/ui.go#L31

• Creating a new folder called web in the prometheusreceiver folder, copying over most of the golang code and html/css/js files but customizing the code to fix the above issues by:

  • Changing the code so that only including the Target, Service Discovery, and Configuration pages are served and shown on the webpage
  • Changing the code to embed the html templates and static files so that they can be built in with the opentelemetry collector executable.

  However this means that a lot of the same code from the prometheus web package is copied over, including the html/css/js files.

[dashpole]

It is similar in concept to zpages, but for prometheus metrics. Having it as an extension like zpages would be nice for decoupling, but looking at the zpages extension, it isn't actually very decoupled, since it is mostly implemented by the service itself, rather than the extension.

I'd lean towards using as much of the upstream prometheus code as we can. It may also be possible to request small changes in upstream prometheus if that would make things much easier for us.

[juliusv]

Hi! @roidelapluie helpfully pointed me at this issue.

Just to let you know, there are two implementations of the UI in Prometheus:

• The legacy UI that is using Go-side HTML template rendering and which you have to explicitly switch to in the UI, so it's not normally used anymore. This is the one you were referring to, but the long-term plan is still to remove this. The only reason why it still exists is that we haven't gotten certain pages as performant yet for large data as the old UI.
• The new React UI, which is the default now. This is a single-page React app (living under https://github.com/prometheus/prometheus/tree/main/web/ui/react-app) that loads all its data via Prometheus's HTTP APIs: https://prometheus.io/docs/prometheus/latest/querying/api/#http-api. We are actually gradually working on making parts of this UI more reusable (@nexucis is doing work on this too), because projects like Cortex and Thanos are having the same UI reusability problem as you have. So it could be an option to work towards a reusable React npm package that allows you to instantiate different parts of the UI. That is, if you are fine using React on the frontend, of course.

[gracewehner]

Thanks @juliusv! I have it working with the React UI now with the same code as above that's using the Prometheus web package. Is there a place where the UI reusability work is tracked or a design proposal? I'd be happy to help with contributing this.

[juliusv]

@gracewehner We don't have a properly tracked effort for it yet on the Prometheus side, as it's more of a loose aspiration of a couple of folks so far who are too distracted by other stuff as well :) Thanos did touch on this a bit in https://github.com/thanos-io/thanos/issues/2198 because it would make sense to reuse components there.

However, @Nexucis already laid some of the groundwork in Prometheus recently by breaking up the single React app into a multi-module structure all within the same repo (all living under https://github.com/prometheus/prometheus/tree/main/web/ui). The only module that is broken out so far is the PromQL text editor support code, but ultimately it could make sense to break out other reusable components in a similar way, so that external consumers like Thanos, Cortex, the Prometheus Receiver, etc. can just "plug together" a partial Prometheus UI that fits their needs. TBH I haven't fully thought through all the implications of that yet in terms of maintainability, reusability, and so on, so I guess this project is looking for someone to lead it :)

[Nexucis]

breaking up the single React

so accurate haha.

joke appart, I'm currently looking to good a candidate that will ultimately validate this new model. codemirror-promql was a special case that cannot be really followed for other potentiel components.

I proposed to start with the Graph page here: https://github.com/prometheus/prometheus/issues/8685. But perhaps it's not the best candidate.

Like by looking at this issue, perhaps sharing the way to display the targets could be more interesting. (and let's be crazy, it could be the good opportunity to tackle the optimization issues around this page)

[philchia]

Is there any updates here?

[cforce]

Seem like the failed test from https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/23244 need to be fixed + approval and as well the MR https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/29622 need reopen/rebase and testing.