Open numonedad opened 2 years ago
I am seeing the same issue on a GKE cluster. I see a list of inactive metrics in Cloud Monitoring, but neither of them contain any data.
Relevant Prometheus config (metrics show up in Prometheus):
scrape_configs:
- job_name: serviceMonitor/ingress-nginx/ingress-nginx-controller/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
Versions used:
Args passed to Stackdriver Prometheus Sidecar:
"--stackdriver.project-id=${var.gcp_project_id}",
"--prometheus.wal-directory=${var.data_dir}/wal",
"--stackdriver.kubernetes.location=${var.gke_region}",
"--stackdriver.kubernetes.cluster-name=${var.gke_cluster_name}",
Stackdriver Prometheus Sidcar logs output:
│ level=warn ts=2021-11-12T10:10:28.769Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:10:58.664Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:11:28.688Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:11:58.785Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:12:28.593Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:12:58.701Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:13:28.748Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:13:58.582Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:14:28.577Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:14:58.684Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:15:28.754Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."
Trying to load any metrics into metrics explorer shows a "no data for this time range" in the graph. The metrics are being pushed as I see metrics ingress bytes count is >0. Is there some way to debug the metrics actually being processed and viewable in metrics explorer? One special thing to note, I do not use GKE. I have to untoggle 'show active' to find these metrics keys.
Here's my prometheus config