Open dgdevops opened 4 years ago
I strongly suspect this is due to particular data points causing an unrecoverable error that looks recoverable. This requires some kind of never-succeeding request to explain, but the sidecar logic absolutely can fall into a permanent retry loop and block the WAL reader when this happens. Documented in the downstream repository
https://github.com/lightstep/opentelemetry-prometheus-sidecar/issues/88
also partly mitigated:
https://github.com/lightstep/opentelemetry-prometheus-sidecar/pulls/87
This is the function that never returns:
// sendSamples to the remote storage with backoff for recoverable errors.
func (s *shardCollection) sendSamplesWithBackoff(client StorageClient, samples []*monitoring_pb.TimeSeries) {
backoff := s.qm.cfg.MinBackoff
for {
begin := time.Now()
err := client.Store(&monitoring_pb.CreateTimeSeriesRequest{TimeSeries: samples})
sentBatchDuration.WithLabelValues(s.qm.queueName).Observe(time.Since(begin).Seconds())
if err == nil {
succeededSamplesTotal.WithLabelValues(s.qm.queueName).Add(float64(len(samples)))
return
}
if _, ok := err.(recoverableError); !ok {
level.Warn(s.qm.logger).Log("msg", "Unrecoverable error sending samples to remote storage", "err", err)
break
}
time.Sleep(time.Duration(backoff))
backoff = backoff * 2
if backoff > s.qm.cfg.MaxBackoff {
backoff = s.qm.cfg.MaxBackoff
}
}
failedSamplesTotal.WithLabelValues(s.qm.queueName).Add(float64(len(samples)))
}
I see the same behaviour with the same messages from the stack-driver sidecar
level=debug ts=2021-02-09T07:25:54.294Z caller=queue_manager.go:306 component=queue_manager msg=QueueManager.calculateDesiredShards samplesIn=0.00173154100250915 samplesOut=0.00173154100250915 samplesOutDuration=5557.854004867412 timePerSample=3.2097732579324483e+06 sizeRate=4890.771316463715 offsetRate=2.134860677902194 desiredShards=0.019098805764792753
level=debug ts=2021-02-09T07:25:54.294Z caller=queue_manager.go:317 component=queue_manager msg=QueueManager.updateShardsLoop lowerBound=0.7 desiredShards=0.019098805764792753 upperBound=1.1
I am using service monitor k8s resources to add targets to Prometheus. I keep receiving metrics in Stackdriver from the sidecar until I add a service monitor to my k8s cluster that adds 220 targets to my prometheus, once the targets come up ALL metrics in stackdriver stop at the same time and no new metric values appear in Stackdriver. Based on the sidecar container logs shard calculation takes place :
This keeps going for hours and hours but the metrics do not return to Stackdriver. Could you please help in understanding the sharding? Additionally, how could I speed up the process?
Thanks