grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.93k stars 3.45k forks source link

error: rpc error: code = Code(500) desc = too many unhealthy instances in the ring #11775

Closed chesha1 closed 2 months ago

chesha1 commented 9 months ago

I am trying to install loki-stack in k8s with helm, the pods are running and ready, but I cannot find any logs in grafana web visualization page.
kubetctl logs loki-0 -n loki shows the following messages:

level=debug ts=2024-01-25T07:56:55.136072801Z caller=grpc_logging.go:46 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=52.279µs msg="gRPC (success)"
level=error ts=2024-01-25T07:56:55.136151371Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n"
...
level=error ts=2024-01-25T07:56:55.136636982Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n"
...
level=error ts=2024-01-25T07:56:55.13809032Z caller=stats.go:57 component=frontend msg="failed to record query metrics" err="expected one of the *LokiRequest, *LokiInstantRequest, *LokiSeriesRequest, *LokiLabelNamesRequest, got "

Below is the initial yaml file I use to helm install loki-stack:

grafana:
  enabled: true
  image:
    registry: docker.mirrors.sjtug.sjtu.edu.cn
    tag: 8.3.5

promtail:
  config:
    clients:
      - url: http://loki:3100/loki/api/v1/push
    file: |
      server:
        log_level: {{ .Values.config.logLevel }}
        log_format: {{ .Values.config.logFormat }}
        http_listen_port: {{ .Values.config.serverPort }}
        {{- with .Values.httpPathPrefix }}
        http_path_prefix: {{ . }}
        {{- end }}
        {{- tpl .Values.config.snippets.extraServerConfigs . | nindent 2 }}

      clients:
        {{- tpl (toYaml .Values.config.clients) . | nindent 2 }}

      positions:
        {{- tpl (toYaml .Values.config.positions) . | nindent 2 }}

      scrape_configs:
        {{- tpl .Values.config.snippets.scrapeConfigs . | nindent 2 }}
        {{- tpl .Values.config.snippets.extraScrapeConfigs . | nindent 2 }}

      limits_config:
        {{- tpl .Values.config.snippets.extraLimitsConfig . | nindent 2 }}

      tracing:
        enabled: {{ .Values.config.enableTracing }}

Below is the config yaml file:

schema_config:
  configs:
    - from: 2020-05-15
      store: boltdb-shipper
      object_store: bos
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/index
    cache_location: /tmp/loki/index_cache
    shared_store: bos

  bos:
    bucket_name: l
    endpoint: bj.bcebos.com
    access_key_id: A
    secret_access_key: 4

compactor:
  working_directory: /tmp/loki/compactor
  shared_store: bos

server:
  log_level: "debug"
  http_listen_port: 3100

ingester:
  wal:
    dir: "/tmp/wal"

auth_enabled: false

common:
  ring:
    kvstore:
      store: "inmemory"

I use the following commands to change config:

kubectl -n loki create secret generic loki --from-file=./loki.yaml -o yaml --dry-run=client | kubectl apply -f -
kubectl -n loki rollout restart statefulset loki

Environment: loki version: 2.9(latest) object storage used: baidu object storage

It would be appreciated if anyone can help. Thank you.

JStickler commented 9 months ago

Questions have a better chance of being answered if you ask them on the community forums.