grafana / helm-charts

Apache License 2.0
1.63k stars 2.26k forks source link

POST /api/v1/push (500): DoBatch: InstancesCount <= 0 #2457

Open dorkamotorka opened 1 year ago

dorkamotorka commented 1 year ago

While trying to send data to Mimir, running on GKE Autopilot, I get the following error (If I do kubectl logs <mimir-distributor-container>):

ts=2023-06-11T08:22:16.595603578Z caller=logging.go:86 level=warn traceID=4815871e105dbd85 msg="POST /api/v1/push (500) 1.800583ms Response: \"DoBatch: InstancesCount <= 0\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 11662; Content-Type: application/x-protobuf; User-Agent: GrafanaAgent/v0.33.2; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: anonymous; "

I'm unable to debug what's going on - the only thing that I'm assuming is that Distributer containers are unable to find Ingester containers.

Here's the helm chart config:

mimir:
  config: |
    usage_stats:
      installation_mode: helm

    activity_tracker:
      filepath: /active-query-tracker/activity.log

    common:
      storage:
        backend: gcs

    blocks_storage:
      gcs:
        bucket_name: {{ .Values.blocks_bucket_name }}

    alertmanager_storage:
      gcs:
        bucket_name: {{ .Values.alert_bucket_name }}

    ruler_storage:
      gcs:
        bucket_name: {{ .Values.ruler_bucket_name }}

compactor:
  containerSecurityContext:
    readOnlyRootFilesystem: false
    runAsNonRoot: false
    runAsUser: 0
ingester:
  zoneAwareReplication:
    enabled: false
  containerSecurityContext:
    readOnlyRootFilesystem: false
    runAsNonRoot: false
    runAsUser: 0
alertmanager:
  containerSecurityContext:
    readOnlyRootFilesystem: false
    runAsNonRoot: false
    runAsUser: 0
ruler:
  containerSecurityContext:
    readOnlyRootFilesystem: false
    runAsNonRoot: false
    runAsUser: 0
store_gateway:
  zoneAwareReplication:
    enabled: false
  containerSecurityContext:
    readOnlyRootFilesystem: false
    runAsNonRoot: false
    runAsUser: 0
minio:
  enabled: false

Any thoughts why this is occuring?

koroglusaban commented 11 months ago

is there any updates, i got same issue