search

Stackdriver / stackdriver-prometheus-sidecar

A sidecar for the Prometheus server that can send metrics to Stackdriver.
https://cloud.google.com/monitoring/kubernetes-engine/prometheus
Apache License 2.0
120 stars 43 forks source link

corruption after 0 bytes: read first header byte: open next segment: next segment 0 too high, expected 1 #212

Closed pjocke closed 4 years ago

pjocke commented 4 years ago

Hi.

I have deployed Prometheus 2.2.1 to GKE from GCP marketplace, and then followed the README in this repo to also deploy the sidecar. The sidecar starts up just fine, but keeps crashing with the following error:

corruption after 0 bytes: read first header byte: open next segment: next segment 0 too high, expected 1

My pods are picked up by the service discovery and gets scraped just fine, so the TSDB and all that goes with it seems fine. The Prometheus server does not log any errors.

I have tried a couple of different tags from the GCR registry, both 0.5 and 0.6 and even 0.4, but they all fail with the same error, which leads me to believe that it might be an issue that I can fix myself. Any advice would be appreciated!

bmoyles0117 commented 4 years ago

Prometheus 2.2.1 is not listed in our compatibility matrix here: https://github.com/Stackdriver/stackdriver-prometheus-sidecar/blob/master/README.md#compatibility

You will need to install Prometheus 2.5+ to ensure that we're not triaging an issue due to incompatibility across the sidecar and the Prometheus server. Closing this issue for now, as we don't plan to support Prometheus 2.2.1 at this time. Please feel free to open a new issue if you experience the same issue with a newer version of Prometheus.