how we scrape log based metrics

[Coder63]

Hi team ,

I try to scrape custom log based metrics and exporter doesn't do anything..

• No scraping for below args

Args

--monitoring.metrics-type-prefixes=logging.googleapis.com/user/*
--monitoring.metrics-type-prefixes=logging.googleapis.com/user/my-custom-metric

level=info ts=2022-01-05T19:47:53.425Z caller=stackdriver_exporter.go:164 msg="Starting stackdriver_exporter" version="(version=0.11.0, branch=HEAD, revision=aafa2f0e851f2e07772a388dbf4590484513a19f)"
3
level=info ts=2022-01-05T19:47:53.425Z caller=stackdriver_exporter.go:165 msg="Build context" build_context="(go=go1.15.1, user=root@67413d69d3db, date=20200902-14:39:30)"
2
level=info ts=2022-01-05T19:47:53.425Z caller=stackdriver_exporter.go:166 msg="Using Google Cloud Project ID" projectID=<ProjectID >
1
level=info ts=2022-01-05T19:47:53.425Z caller=stackdriver_exporter.go:188 msg="Listening on" address=:9255

• Scraper pods crashes with duplicate metric name error for below args

Args

--monitoring.metrics-type-prefixes=logging.googleapis.com
--monitoring.metrics-type-prefixes=logging.googleapis.com/user

panic: duplicate label names goroutine 128 [running]: github.com/prometheus/client_golang/prometheus.MustNewConstHistogram(...) /app/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:620 github.com/prometheus-community/stackdriver_exporter/collectors.(*TimeSeriesMetrics).newConstHistogram(0xc00050fa08, 0xc0002e8000, 0x4d, 0x1a3388e8, 0xed967ec17, 0x0, 0xc000686100, 0xb, 0x10, 0xc0001e9440, ...) /app/collectors/monitoring_metrics.go:102 +0x228 github.com/prometheus-community/stackdriver_exporter/collectors.(*TimeSeriesMetrics).completeHistogramMetrics(0xc00050fa08) /app/collectors/monitoring_metrics.go:199 +0x1f0 github.com/prometheus-community/stackdriver_exporter/collectors.(*TimeSeriesMetrics).Complete(0xc00050fa08) /app/collectors/monitoring_metrics.go:162 +0x39 github.com/prometheus-community/stackdriver_exporter/collectors.(*MonitoringCollector).reportTimeSeriesMetrics(0xc00027e300, 0xc0001e8bd0, 0xc000770800, 0xc0000b78c0, 0xc0001e8bd0, 0x0) /app/collectors/monitoring_collector.go:404 +0x1465 github.com/prometheus-community/stackdriver_exporter/collectors.(*MonitoringCollector).reportMonitoringMetrics.func1.1(0xc0004aea50, 0xc00027e300, 0x1a338c07, 0xed967eb63, 0x0, 0x1a338c07, 0xed967ec8f, 0x0, 0xc000731a40, 0xc000770800, ...) /app/collectors/monitoring_collector.go:257 +0x6b6 created by github.com/prometheus-community/stackdriver_exporter/collectors.(*MonitoringCollector).reportMonitoringMetrics.func1 /app/collectors/monitoring_collector.go:231 +0x3f5

Can someone help on this one.. Thanks

[Coder63]

Any moderators up here ??

[weyert]

For me it looked like they end up under stackdriver_k_8_s_container_logging_googleapis_com_user_???

[romankor]

@Coder63 did you figure it out ?

[jeremyp-synopsys]

For anyone that stumbles onto this, the method that I used was to use a metrics-type-prefix of logging.googleapis.com/user/ which collected all of the log-based metrics defined within the project.

The metric name is based on the underlying resource associated with the log. For example a log-based metric named foo-bar against logs from a cloud function resulted in a prom metric named stackdriver_cloud_function_logging_googleapis_com_user_foo_bar