search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.94k stars 3.45k forks source link

Unavailable desc = name resolver error: produced zero addresses #8361

Open patsevanton opened 1 year ago

patsevanton commented 1 year ago

Describe the bug

loki-loki-distributed-querier-668f5f9954-vq8vh querier level=warn ts=2023-02-01T00:39:22.03008595Z caller=logging.go:86 traceID=5612ff99f8d6cf85 orgID=fake msg="GET /loki/api/v1/labels?end=1675211962001000000&start=1675211362001000000 (500) 381.309µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 92.883µs; X-Scope-Orgid: fake; uber-trace-id: 5612ff99f8d6cf85:1a2ea48e787d98d6:0654888a630c93e8:0; "

loki-loki-distributed-query-frontend-64944df784-7vtsh query-frontend level=error ts=2023-02-01T00:39:22.025418368Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"

To Reproduce Steps to reproduce the behavior:

  1. Started Loki: 2.7.1
  2. Query: {} term ??

Expected behavior Connect and work loki datasource in grafana

Environment:

Loki config

compactor:
  enabled: true
  persistence:
    enabled: true
    storageClass: yc-network-ssd
  resources:
    limits:
      cpu: 200m
      memory: 384Mi
    requests:
      cpu: 100m
      memory: 256Mi
distributor:
  autoscaling:
    enabled: true
    maxReplicas: 4
    minReplicas: 2
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 80
  maxUnavailable: 1
  resources:
    limits:
      cpu: 200m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi
gateway:
  enabled: false
global:
  image:
    registry: harbor.corp/dockerhub
indexGateway:
  enabled: true
  maxUnavailable: 1
  persistence:
    enabled: true
    storageClass: yc-network-ssd
  resources:
    limits:
      cpu: 200m
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 64Mi
ingester:
  affinity: {}
  autoscaling:
    enabled: true
    maxReplicas: 12
    minReplicas: 5
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 80
  maxUnavailable: 1
  persistence:
    enabled: true
    storageClass: yc-network-ssd
  resources:
    limits:
      cpu: 600m
      memory: 4096Mi
    requests:
      cpu: 100m
      memory: 1512Mi
ingress:
  annotations:
    nginx.ingress.kubernetes.io/client-body-timeout: "610"
    nginx.ingress.kubernetes.io/client-header-timeout: "610"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "610"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "610"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "610"
  enabled: true
  hosts:
  - loki-k8s.corp
  ingressClassName: nginx
  paths:
    distributor:
    - /api/prom/push
    - /loki/api/v1/push
    querier:
    - /api/prom/tail
    - /loki/api/v1/tail
    query-frontend:
    - /loki/api
    ruler:
    - /api/prom/rules
    - /loki/api/v1/rules
    - /prometheus/api/v1/rules
    - /prometheus/api/v1/alerts
loki:
  config: |
    auth_enabled: false
    server:
      http_listen_port: 3100
      http_server_read_timeout: 610s
      http_server_write_timeout: 610s
    common:
      compactor_address: http://{{ include "loki.compactorFullname" . }}:3100
    distributor:
      ring:
        kvstore:
          store: memberlist
    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist
    ingester:
      autoforget_unhealthy: true
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 30m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 20m
      max_transfer_retries: 0
      wal:
        dir: /var/loki/wal
    limits_config:
      ingestion_rate_mb: 10000
      ingestion_burst_size_mb: 1000
      max_global_streams_per_user: 10000
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      split_queries_by_interval: 24h
      retention_period: 240h
    {{- if .Values.loki.schemaConfig}}
    schema_config:
    {{- toYaml .Values.loki.schemaConfig | nindent 2}}
    {{- end}}
    {{- if .Values.loki.storageConfig}}
    storage_config:
    {{- if .Values.indexGateway.enabled}}
    {{- $indexGatewayClient := dict "server_address" (printf "dns:///%s:9095" (include "loki.indexGatewayFullname" .)) }}
    {{- $_ := set .Values.loki.storageConfig.boltdb_shipper "index_gateway_client" $indexGatewayClient }}
    {{- end}}
    {{- toYaml .Values.loki.storageConfig | nindent 2}}
    {{- if .Values.memcachedIndexQueries.enabled }}
      index_queries_cache_config:
        memcached:
          batch_size: 100
          parallelism: 100
        memcached_client:
          consistent_hash: true
          host: {{ include "loki.memcachedIndexQueriesFullname" . }}
          service: memcached-client
    {{- end}}
    {{- end}}

    runtime_config:
      file: /var/{{ include "loki.name" . }}-runtime/runtime.yaml
    chunk_store_config:
      max_look_back_period: 240h
    table_manager:
      retention_deletes_enabled: false
      retention_period: 0s
    querier:
      query_timeout: 600s
      engine:
        timeout: 600s
    query_range:
      align_queries_with_step: true
      max_retries: 5
      cache_results: true
      results_cache:
        cache:
          {{- if .Values.memcachedFrontend.enabled }}
          memcached_client:
            consistent_hash: true
            host: {{ include "loki.memcachedFrontendFullname" . }}
            max_idle_conns: 16
            service: memcached-client
            timeout: 500ms
            update_interval: 1m
          {{- else }}
          embedded_cache:
            enabled: true
            ttl: 24h
          {{- end }}
    frontend_worker:
      {{- if .Values.queryScheduler.enabled }}
      scheduler_address: {{ include "loki.querySchedulerFullname" . }}:9095
      {{- else }}
      frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
      {{- end }}
    frontend:
      log_queries_longer_than: 5s
      compress_responses: true
      {{- if .Values.queryScheduler.enabled }}
      scheduler_address: {{ include "loki.querySchedulerFullname" . }}:9095
      {{- end }}
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100
    compactor:
      shared_store: s3
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: https://alertmanager.xx
      external_url: https://alertmanager.xx
  structuredConfig:
    ingester:
      chunk_idle_period: 1h
      chunk_target_size: 1536000
      max_chunk_age: 2h
      max_transfer_retries: 0
    schema_config:
      configs:
      - from: "2020-09-07"
        index:
          period: 24h
          prefix: loki_index_
        object_store: s3
        schema: v11
        store: boltdb-shipper
      - from: "2023-02-01"
        index:
          period: 24h
          prefix: tsdb_index_
        object_store: s3
        schema: v12
        store: tsdb
    storage_config:
      aws:
        bucketnames: core-dev-loki
        s3: https://xxxxxxxxx@storage.cloud.net/
      boltdb_shipper:
        cache_ttl: 168h
        shared_store: s3
      tsdb_shipper:
        active_index_directory: /var/loki/tsdb-index
        cache_location: /var/loki/tsdb-cache
        index_gateway_client:
          server_address: dns:///index-gateway:9095
        query_ready_num_days: 7
        shared_store: s3
memcached:
  image:
    repository: library/memcached
memcachedIndexQueries:
  enabled: true
  maxUnavailable: 1
  replicas: 3
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 50m
      memory: 64Mi
querier:
  autoscaling:
    enabled: true
    maxReplicas: 20
    minReplicas: 6
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 80
  maxUnavailable: 1
  resources:
    limits:
      cpu: 2000m
      memory: 3096Mi
    requests:
      cpu: 200m
      memory: 3096Mi
queryFrontend:
  affinity: {}
  autoscaling:
    enabled: true
    maxReplicas: 6
    minReplicas: 3
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: 80
  extraArgs:
  - -log.level=debug
  maxUnavailable: 1
  resources:
    limits:
      cpu: 200m
      memory: 512Mi
    requests:
      cpu: 150m
      memory: 256Mi
queryScheduler:
  enabled: true
serviceMonitor:
  annotations:
    prometheus.io/port: "3100"
    prometheus.io/scrape: "true"
  enabled: true
  labels:
    release: prometheus-operator
patsevanton commented 1 year ago

Add log.level=debug for queryScheduler, querier

Debug quiery

loki-loki-distributed-query-scheduler-f6d8d7dd-5cn6r query-scheduler level=debug ts=2023-02-01T02:11:12.319053413Z caller=logging.go:76 traceID=34f46c9de2f9dc82 orgID=fake msg="GET /ready (200) 44.14µs"
loki-loki-distributed-querier-c7cf896c4-g54qt querier level=debug ts=2023-02-01T02:11:12.389776118Z caller=logging.go:76 traceID=06469fc5fad5851d orgID=fake msg="GET /ready (200) 52.811µs"
loki-loki-distributed-querier-c7cf896c4-g54qt querier level=debug ts=2023-02-01T02:11:12.389776111Z caller=logging.go:76 traceID=4af172f9a3ed8778 orgID=fake msg="GET /ready (200) 52.81µs"
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661477481Z caller=spanlogger.go:80 user=fake method=query.Label level=warn msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." call=WrapQuerySpanAndTimeout
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier level=warn ts=2023-02-01T02:11:12.661524676Z caller=querier.go:361 msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." err=null call=Label
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661617309Z caller=spanlogger.go:80 user=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661728421Z caller=spanlogger.go:80 user=fake level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661764581Z caller=spanlogger.go:80 user=fake level=debug Cache.Chunk.Requests=0 Cache.Chunk.EntriesRequested=0 Cache.Chunk.EntriesFound=0 Cache.Chunk.EntriesStored=0 Cache.Chunk.BytesSent="0 B" Cache.Chunk.BytesReceived="0 B" Cache.Index.Requests=0 Cache.Index.EntriesRequested=0 Cache.Index.EntriesFound=0 Cache.Index.EntriesStored=0 Cache.Index.BytesSent="0 B" Cache.Index.BytesReceived="0 B" Cache.Result.Requests=0 Cache.Result.EntriesRequested=0 Cache.Result.EntriesFound=0 Cache.Result.EntriesStored=0 Cache.Result.BytesSent="0 B" Cache.Result.BytesReceived="0 B"
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661786259Z caller=spanlogger.go:80 user=fake level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=197.769µs Summary.QueueTime=0s
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.661806885Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=10m0s duration=197.769µs status=500 label= throughput=0B total_bytes=0B total_entries=0
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier level=warn ts=2023-02-01T02:11:12.661863835Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?end=1675217472607000000&start=1675216872607000000 (500) 443.999µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 39.636µs; X-Scope-Orgid: fake; uber-trace-id: 6f3a8942248c05e0:3ea308ea05e015fd:6e943c4002fe04b7:0; "
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.665152076Z caller=spanlogger.go:80 user=fake method=query.Label level=warn msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." call=WrapQuerySpanAndTimeout
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier level=warn ts=2023-02-01T02:11:12.665188276Z caller=querier.go:361 msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." err=null call=Label
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.665274967Z caller=spanlogger.go:80 user=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.665351637Z caller=spanlogger.go:80 user=fake level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.665382721Z caller=spanlogger.go:80 user=fake level=debug Cache.Chunk.Requests=0 Cache.Chunk.EntriesRequested=0 Cache.Chunk.EntriesFound=0 Cache.Chunk.EntriesStored=0 Cache.Chunk.BytesSent="0 B" Cache.Chunk.BytesReceived="0 B" Cache.Index.Requests=0 Cache.Index.EntriesRequested=0 Cache.Index.EntriesFound=0 Cache.Index.EntriesStored=0 Cache.Index.BytesSent="0 B" Cache.Index.BytesReceived="0 B" Cache.Result.Requests=0 Cache.Result.EntriesRequested=0 Cache.Result.EntriesFound=0 Cache.Result.EntriesStored=0 Cache.Result.BytesSent="0 B" Cache.Result.BytesReceived="0 B"
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.665403372Z caller=spanlogger.go:80 user=fake level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=158.385µs Summary.QueueTime=0s
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier ts=2023-02-01T02:11:12.66542977Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=10m0s duration=158.385µs status=500 label= throughput=0B total_bytes=0B total_entries=0
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier level=warn ts=2023-02-01T02:11:12.665471793Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?end=1675217472607000000&start=1675216872607000000 (500) 369.013µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 30.544µs; X-Scope-Orgid: fake; uber-trace-id: 6f3a8942248c05e0:3ea308ea05e015fd:6e943c4002fe04b7:0; "
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.652836279Z caller=spanlogger.go:80 user=fake method=query.Label level=warn msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." call=WrapQuerySpanAndTimeout
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier level=warn ts=2023-02-01T02:11:12.65288613Z caller=querier.go:361 msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." err=null call=Label
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.652972891Z caller=spanlogger.go:80 user=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.653079366Z caller=spanlogger.go:80 user=fake level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=error ts=2023-02-01T02:11:12.646450502Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=error ts=2023-02-01T02:11:12.649343345Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=error ts=2023-02-01T02:11:12.652014774Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=error ts=2023-02-01T02:11:12.653303485Z caller=retry.go:73 org_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=error ts=2023-02-01T02:11:12.654596185Z caller=retry.go:73 org_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki-loki-distributed-query-frontend-5c78d64968-6hgqg query-frontend level=warn ts=2023-02-01T02:11:12.65465888Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?start=1675216872607000000&end=1675217472607000000 (500) 11.171432ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Grafana/9.1.6; X-Forward-For: 10.236.30.21; X-Forwarded-For: 10.236.30.21; X-Forwarded-Host: loki-k8s.mos.corp; X-Forwarded-Port: 80; X-Forwarded-Proto: http; X-Forwarded-Scheme: http; X-Real-Ip: 10.236.30.21; X-Request-Id: 9e0559641ea5ce75b841be9962ada664; X-Scheme: http; "
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.653113849Z caller=spanlogger.go:80 user=fake level=debug Cache.Chunk.Requests=0 Cache.Chunk.EntriesRequested=0 Cache.Chunk.EntriesFound=0 Cache.Chunk.EntriesStored=0 Cache.Chunk.BytesSent="0 B" Cache.Chunk.BytesReceived="0 B" Cache.Index.Requests=0 Cache.Index.EntriesRequested=0 Cache.Index.EntriesFound=0 Cache.Index.EntriesStored=0 Cache.Index.BytesSent="0 B" Cache.Index.BytesReceived="0 B" Cache.Result.Requests=0 Cache.Result.EntriesRequested=0 Cache.Result.EntriesFound=0 Cache.Result.EntriesStored=0 Cache.Result.BytesSent="0 B" Cache.Result.BytesReceived="0 B"
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.653132382Z caller=spanlogger.go:80 user=fake level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=189.413µs Summary.QueueTime=0s
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.653158075Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=10m0s duration=189.413µs status=500 label= throughput=0B total_bytes=0B total_entries=0
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier level=warn ts=2023-02-01T02:11:12.653204739Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?end=1675217472607000000&start=1675216872607000000 (500) 418.05µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 55.613µs; X-Scope-Orgid: fake; uber-trace-id: 6f3a8942248c05e0:3ea308ea05e015fd:6e943c4002fe04b7:0; "
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.660911343Z caller=spanlogger.go:80 user=fake method=query.Label level=warn msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." call=WrapQuerySpanAndTimeout
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier level=warn ts=2023-02-01T02:11:12.660941466Z caller=querier.go:361 msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." err=null call=Label
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.661024292Z caller=spanlogger.go:80 user=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.661101755Z caller=spanlogger.go:80 user=fake level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.661132176Z caller=spanlogger.go:80 user=fake level=debug Cache.Chunk.Requests=0 Cache.Chunk.EntriesRequested=0 Cache.Chunk.EntriesFound=0 Cache.Chunk.EntriesStored=0 Cache.Chunk.BytesSent="0 B" Cache.Chunk.BytesReceived="0 B" Cache.Index.Requests=0 Cache.Index.EntriesRequested=0 Cache.Index.EntriesFound=0 Cache.Index.EntriesStored=0 Cache.Index.BytesSent="0 B" Cache.Index.BytesReceived="0 B" Cache.Result.Requests=0 Cache.Result.EntriesRequested=0 Cache.Result.EntriesFound=0 Cache.Result.EntriesStored=0 Cache.Result.BytesSent="0 B" Cache.Result.BytesReceived="0 B"
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.661151402Z caller=spanlogger.go:80 user=fake level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=154.873µs Summary.QueueTime=0s
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier ts=2023-02-01T02:11:12.661175111Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=10m0s duration=154.873µs status=500 label= throughput=0B total_bytes=0B total_entries=0
loki-loki-distributed-querier-c7cf896c4-tzgp4 querier level=warn ts=2023-02-01T02:11:12.661215571Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?end=1675217472607000000&start=1675216872607000000 (500) 344.627µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 32.026µs; X-Scope-Orgid: fake; uber-trace-id: 6f3a8942248c05e0:3ea308ea05e015fd:6e943c4002fe04b7:0; "
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657222718Z caller=spanlogger.go:80 user=fake method=query.Label level=warn msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." call=WrapQuerySpanAndTimeout
loki-loki-distributed-querier-c7cf896c4-g54qt querier level=warn ts=2023-02-01T02:11:12.657269067Z caller=querier.go:361 msg="deprecated querier:query_timeout YAML configuration identified. Please migrate to limits:query_timeout instead." err=null call=Label
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657366234Z caller=spanlogger.go:80 user=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657480968Z caller=spanlogger.go:80 user=fake level=debug Ingester.TotalReached=0 Ingester.TotalChunksMatched=0 Ingester.TotalBatches=0 Ingester.TotalLinesSent=0 Ingester.TotalChunksRef=0 Ingester.TotalChunksDownloaded=0 Ingester.ChunksDownloadTime=0s Ingester.HeadChunkBytes="0 B" Ingester.HeadChunkLines=0 Ingester.DecompressedBytes="0 B" Ingester.DecompressedLines=0 Ingester.CompressedBytes="0 B" Ingester.TotalDuplicates=0 Querier.TotalChunksRef=0 Querier.TotalChunksDownloaded=0 Querier.ChunksDownloadTime=0s Querier.HeadChunkBytes="0 B" Querier.HeadChunkLines=0 Querier.DecompressedBytes="0 B" Querier.DecompressedLines=0 Querier.CompressedBytes="0 B" Querier.TotalDuplicates=0
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657518639Z caller=spanlogger.go:80 user=fake level=debug Cache.Chunk.Requests=0 Cache.Chunk.EntriesRequested=0 Cache.Chunk.EntriesFound=0 Cache.Chunk.EntriesStored=0 Cache.Chunk.BytesSent="0 B" Cache.Chunk.BytesReceived="0 B" Cache.Index.Requests=0 Cache.Index.EntriesRequested=0 Cache.Index.EntriesFound=0 Cache.Index.EntriesStored=0 Cache.Index.BytesSent="0 B" Cache.Index.BytesReceived="0 B" Cache.Result.Requests=0 Cache.Result.EntriesRequested=0 Cache.Result.EntriesFound=0 Cache.Result.EntriesStored=0 Cache.Result.BytesSent="0 B" Cache.Result.BytesReceived="0 B"
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657541357Z caller=spanlogger.go:80 user=fake level=debug Summary.BytesProcessedPerSecond="0 B" Summary.LinesProcessedPerSecond=0 Summary.TotalBytesProcessed="0 B" Summary.TotalLinesProcessed=0 Summary.ExecTime=201.228µs Summary.QueueTime=0s
loki-loki-distributed-querier-c7cf896c4-g54qt querier ts=2023-02-01T02:11:12.657571836Z caller=spanlogger.go:80 user=fake level=info org_id=fake latency=fast query_type=labels length=10m0s duration=201.228µs status=500 label= throughput=0B total_bytes=0B total_entries=0
loki-loki-distributed-querier-c7cf896c4-g54qt querier level=warn ts=2023-02-01T02:11:12.657631305Z caller=logging.go:86 traceID=6f3a8942248c05e0 orgID=fake msg="GET /loki/api/v1/labels?end=1675217472607000000&start=1675216872607000000 (500) 457.6µs Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Query-Queue-Time: 48.252µs; X-Scope-Orgid: fake; uber-trace-id: 6f3a8942248c05e0:3ea308ea05e015fd:6e943c4002fe04b7:0; "
loki-loki-distributed-query-scheduler-f6d8d7dd-2fpm6 query-scheduler level=debug ts=2023-02-01T02:11:12.718654397Z caller=logging.go:76 traceID=5c2cc5734965a2e4 orgID=fake msg="GET /ready (200) 40.772µs"
loki-loki-distributed-query-scheduler-f6d8d7dd-2fpm6 query-scheduler level=debug ts=2023-02-01T02:11:12.7186546Z caller=logging.go:76 traceID=09503adb2816b5c4 orgID=fake msg="GET /ready (200) 40.844µs"
loki-loki-distributed-querier-c7cf896c4-qdf7d querier level=debug ts=2023-02-01T02:11:13.080733022Z caller=logging.go:76 traceID=76daf41ce579feab orgID=fake msg="GET /metrics (200) 4.346942ms"
loki-loki-distributed-querier-c7cf896c4-snvj8 querier level=debug ts=2023-02-01T02:11:13.19509238Z caller=logging.go:76 traceID=4f273c80e206f4d8 orgID=fake msg="GET /ready (200) 55.569µs"
loki-loki-distributed-querier-c7cf896c4-snvj8 querier level=debug ts=2023-02-01T02:11:13.195120513Z caller=logging.go:76 traceID=7deb494367fc3963 orgID=fake msg="GET /ready (200) 46.242µs"
loki-loki-distributed-querier-c7cf896c4-v2px9 querier level=debug ts=2023-02-01T02:11:13.526619717Z caller=logging.go:76 traceID=697085d001219976 orgID=fake msg="GET /ready (200) 48.644µs"
loki-loki-distributed-querier-c7cf896c4-v2px9 querier level=debug ts=2023-02-01T02:11:13.526701317Z caller=logging.go:76 traceID=79a43900ae4b0d57 orgID=fake msg="GET /ready (200) 43.025µs"
loki-loki-distributed-querier-c7cf896c4-qdf7d querier level=debug ts=2023-02-01T02:11:14.00756914Z caller=logging.go:76 traceID=2145ac162d71e484 orgID=fake msg="GET /ready (200) 56.547µs"
loki-loki-distributed-querier-c7cf896c4-vgmj7 querier level=debug ts=2023-02-01T02:11:14.373540641Z caller=logging.go:76 traceID=4d01115f5e34da81 orgID=fake msg="GET /ready (200) 57.324µs"
loki-loki-distributed-querier-c7cf896c4-qdf7d querier ts=2023-02-01T02:11:15.333116118Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: loki-loki-distributed-compactor-5846db8f7d-75pcn-1098edcd 10.236.164.29:7946"
DylanGuedes commented 1 year ago

Hey, thanks for your report.

This is probably an environment problem and not exactly a Loki problem, but anyway, a few suggestions:

Anindya-Banerjee commented 1 year ago

I would like to report I face the exact same issue after switching to tsdb.

I used version 0.69.6 of the loki-distributed helm chart to deploy Loki and switched from boltdb-shipper to tsdb in the config. Log ingestion works fine and there are no errors in the ingestors. However, when i try to query logs that were ingested using tsdb-shipper the qurier returns a msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n" However, when i query historical logs that were ingested using boltdb-shipper, the querier works just fine and doesn't throw any errors.

Do let me know if i can provide you with any further details that might help with the investigation.

TheiLLeniumStudios commented 1 year ago

@Anindya-Banerjee Make sure to use dns://loki-index-gateway:9095 as the index_gateway_client.server_address under tsdb_shipper. Or if your service name is different for index-gateway, use that. You can check it by running kubectl get svc -n <loki-namespace> | grep index-gateway