search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.39k stars 203 forks source link

ability to expose metrics from multiple integrations on a /metrics endpoint #576

Open rohankmr414 opened 6 months ago

rohankmr414 commented 6 months ago

Request

I want to use grafana alloy for exposing node exporter and process exporter metrics, in the documentation I can see that there's a way to forward those metrics to a prometheus remote write url, I wanted to know if there's a way to expose those metrics on a /metrics type endpoint in grafana alloy.

Use case

We want to use it in our already existing monitoring system which scrapes metrics from a huge fleet of EC2 instances, we don't want to remote write from every EC2 instance.

tpaschalis commented 6 months ago

The answer is.. this kinda exists today already.

In Alloy's built-in debug UI, you can see the endpoint that each exporter component exposes internally (eg. localhost:12345/api/v0/component/prometheus.exporter.self.LABEL/metrics`).

This uses an unversioned and undocumented API, so we should find a way to do the same more robustly.

(edit: I've amended this comment after seeing that the next comment and the discussion that'd already taken place in the community slack).

rohankmr414 commented 6 months ago

@tpaschalis I was looking for an option to configure a single endpoint for serving multiple integrations, just like in fluent bit where you can configure what needs to be exposed. @rfratto mentioned in the community slack that the stability of this API is not guaranteed.

tpaschalis commented 6 months ago

Yeah, you're right, I see, thanks. I've amended the comment above.

github-actions[bot] commented 5 months ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

itjobs-levi commented 4 months ago

Hi!

There may be several ways, but in my case, I do not prefer that the collection target server directly puts it into the metric storage. This is because the configuration can change frequently and the collection target server does not have enough resources to configure the Alloy agent for high availability. Also, if collection target server filtering is required, it can be managed centrally from the collector cluster.

(Usually, one Alloy agent will be used for the collection target server.)

Therefore, the collection target server will only perform up to prometheus.*.exporter and configure the collector as a cluster to scrape from multiple servers.

I am scraping by specifying /api/v0/comComponent/prometheus.exporter.[unix|process|...].LABEL/metrics. Of course, it works well.

However, it is said that this is not a recommended method. If that API is not official, I think that alloy should declare the exporter information exposed by the collection target server when scraping and scrape it from the API. Alternatively, multiple exporter metrics should be added to the /metrics base metrics address.

Upanshu11 commented 2 months ago

Any update on this? I am already using Grafana Agent and was looking to migrate to Alloy. On top of it, I also have a use case of HPA, where I want a metrics API to expose my custom metrics for autoscaling. If alloy exposes my custom metrics on /metrics endpoint, it can help me in avoiding setting up a Prom adapter or an alternative. Or does Alloy provide a totally different way to help in that?