Open samjewell opened 1 year ago
The scrape_...
metrics are so-called synthetic metrics. They are not exposed by the target, but they are created by Prometheus itself during scraping based on the scrape. Another synthetic metric (and the most famous of all) is the up
metric.
Metadata is still not a native concept of the internal Prometheus data model. It is only really present in the exposition format (from where it is quite superficially handled and made available via the metadata API). But since synthetic metrics are only created internally and never exposed, they never get metadata attached.
One might argue the metadata should still be created for the sake of the metadata API. To me, that would feel like patching an already quite patchy API even more. It would be much better to finally fully embrace metadata in the internal data model (as it is beginning right now, out of necessity, with native histograms).
To get back to your original concern, you can estimate the scrape interval by counting the number of times the up
metrics were inserted over the last hour with the following query:
count_over_time(up[1h])
This query returns the number of times the targets were scraped over the last hour.
I get that this is 'not as easy as it looks', but can someone confirm the metric type of scrape_samples_scraped
? Am I correct that its a gauge and not a counter?
I'm guessing its a gauge based on this blog post, which says
Ingestion rate can be calculated from scrape_samples_scraped metric exposed by Prometheus using the following PromQL query:
sum_over_time(scrape_samples_scraped[5m]) / 300
Yes it is a gauge
Thanks!
What did you do?
I downloaded and ran the Prometheus binary with the default config to scrape itself.
I found that:
scrape_samples_scraped
at/graph
endpoint/api/v1/metadata
there was no entry forscrape_samples_scraped
I believe the same is true of:
scrape_series_added
scrape_samples_post_metric_relabeling
I'm trying to both:
Within Grafana you can typically see the HELP part of the metric-metadata on hover in the metrics browser:
But for this metric, there's nothing useful:
I'm interested in this metric in particular because I think it can help diagnose problems with people setting their scrape_interval much smaller than they need (and shipping data far too frequently). For example, as described here: https://grafana.com/docs/grafana-cloud/billing-and-usage/control-prometheus-metrics-usage/changing-scrape-interval/#grafana-agent
What did you expect to see?
HELP metadata for this metric both on the /api/v1/metadata endpoint and in Grafana (on hover, in the metrics-browser).
What did you see instead? Under which circumstances?
No metadata
System information
Darwin 21.6.0 arm64
Prometheus version
Prometheus configuration file
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response