Unable to see the logs on grafana dashboard which is getting stored in PVC in loki distributed

[RushikeshChanakhekar]

I am using loki distributed helm chart and configured the PVC to store logs of long duration but I can see only last 3 hrs of data on grafana dashboard.

In ingester logs I can see this

level=info ts=2024-11-14T08:56:06.656395165Z caller=flush.go:167 msg="flushing stream" user=sonarqube fp=d82a7b498cb1cdbf immediate=false num_chunks=1 labels="{app=\"sonarqube-instance-7847\", container=\"database\", filename=\"/var/log/pods/sonarqube_sonarqube-instance-7847-0_a1dd37e4-44da-453b-b371-6da6831a54b1/database/0.log\", job=\"sonarqube/sonarqube-instance-7847\", namespace=\"sonarqube\", node_name=\"devops-dev-worker-7ac1e493-nn2sb\", pod=\"sonarqube-instance-7847-0\", stream=\"stderr\", time=\"2024-11-14T09:26:01.956230139+01:00\"}"

As per my observation, I cant see the data which is flushed on grafana dashboard.

my values.yaml

gateway:
  enabled: false
querier:
  query_timeout: 5m
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""
    annotations: {}
  extraContainers:
    - name: reverse-proxy
      image: k8spin/loki-multi-tenant-proxy:v1.0.0
      args:
        - "run"
        - "--port=3101"
        - "--loki-server=https://<hostname>"
        - "--auth-config=/etc/reverse-proxy-conf/authn.yaml"
      ports:
        - name: http
          containerPort: 3101
          protocol: TCP
      resources:
        limits:
          cpu: 250m
          memory: 200Mi
        requests:
          cpu: 50m
          memory: 40Mi
      volumeMounts:
        - name: reverse-proxy-auth-config
          mountPath: /etc/reverse-proxy-conf
  extraVolumes:
    - name: reverse-proxy-auth-config
      secret:
        secretName: reverse-proxy-auth-config
  extraPorts:
    - port: 3101
      protocol: TCP
      name: http
      targetPort: http
ingester:
  persistence:
    enabled: true
    inMemory: false
    claims:
      - name: data
        size: 400Gi
        storageClass: ""
loki:
  image:
    tag: 2.9.8
  config: |
    auth_enabled: true  
    server:
      {{- toYaml .Values.loki.server | nindent 6 }} 
    common:
      compactor_address: http://{{ include "loki.compactorFullname" . }}:3100
    distributor:
      ring:
        kvstore:
          store: memberlist   
    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist
    ingester_client:
      grpc_client_config:
        grpc_compression: gzip
    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_idle_period: 30m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 1m
      max_transfer_retries: 0
      max_chunk_age: 24h
      wal:
        dir: /var/loki/wal
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      split_queries_by_interval: 15m
      retention_period: 240h
    {{- if .Values.loki.schemaConfig}}
    schema_config:
    {{- toYaml .Values.loki.schemaConfig | nindent 2}}
    {{- end}}
    {{- if .Values.loki.storageConfig}}
    storage_config:
    {{- if .Values.indexGateway.enabled}}
    {{- $indexGatewayClient := dict "server_address" (printf "dns:///%s:9095" (include "loki.indexGatewayFullname" .)) }}
    {{- $_ := set .Values.loki.storageConfig.boltdb_shipper "index_gateway_client" $indexGatewayClient }}
    {{- end}}
    {{- toYaml .Values.loki.storageConfig | nindent 2}}
    {{- if .Values.memcachedIndexQueries.enabled }}
      index_queries_cache_config:
        memcached_client:
          addresses: dnssrv+_memcached-client._tcp.{{ include "loki.memcachedIndexQueriesFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}
          consistent_hash: true
    {{- end}}
    {{- end}}  
    runtime_config:
      file: /var/{{ include "loki.name" . }}-runtime/runtime.yaml
    chunk_store_config:
      max_look_back_period: 0s
      {{- if .Values.memcachedChunks.enabled }}
      chunk_cache_config:
        embedded_cache:
          enabled: false
        memcached_client:
          consistent_hash: true
          addresses: dnssrv+_memcached-client._tcp.{{ include "loki.memcachedChunksFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}
      {{- end }}
      {{- if .Values.memcachedIndexWrites.enabled }}
      write_dedupe_cache_config:
        memcached_client:
          consistent_hash: true
          addresses: dnssrv+_memcached-client._tcp.{{ include "loki.memcachedIndexWritesFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}
      {{- end }}
    table_manager:
      retention_deletes_enabled: true
      retention_period: 240h
    query_range:
      align_queries_with_step: true
      max_retries: 5
      cache_results: true
      parallelise_shardable_queries: true
      results_cache:
        cache:
          {{- if .Values.memcachedFrontend.enabled }}
          memcached_client:
            addresses: dnssrv+_memcached-client._tcp.{{ include "loki.memcachedFrontendFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}
            consistent_hash: true
          {{- else }}
          embedded_cache:
            enabled: true
            ttl: 24h
          {{- end }} 
    frontend_worker:
      {{- if .Values.queryScheduler.enabled }}
      scheduler_address: {{ include "loki.querySchedulerFullname" . }}:9095
      {{- else }}
      frontend_address: {{ include "loki.queryFrontendFullname" . }}-headless:9095
      {{- end }} 
    frontend:
      log_queries_longer_than: 60s
      compress_responses: true
      {{- if .Values.queryScheduler.enabled }}
      scheduler_address: {{ include "loki.querySchedulerFullname" . }}:9095
      {{- end }}
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100 
    compactor:
      shared_store: filesystem
      working_directory: /var/loki/compactor
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: filesystem
    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: https://alertmanager.xx
      external_url: https://alertmanager.xx
    # -- Check https://grafana.com/docs/loki/latest/configuration/#schema_config for more info on how to configure schemas
  schemaConfig:
    configs:
    - from: "2020-09-07"
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
    # -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
  storageConfig:
    boltdb_shipper:
      shared_store: filesystem
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 168h
    filesystem:
      directory: /var/loki/chunks
    # -- Uncomment to configure each storage individually
    #   azure: {}
    #   gcs: {}
    #   s3: {}
    #   boltdb: {}
ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: loki-distributed-basic-auth
    nginx.ingress.kubernetes.io/auth-secret-type: auth-map
  paths:
    distributor:
      - /api/prom/push
      - /loki/api/v1/push
    querier:
      - /api/prom/tail
      - /loki/api/v1/tail
    query-frontend:
      - /loki/api
    ruler:
      - /api/prom/rules
      - /loki/api/v1/rules
      - /prometheus/api/v1/rules
      - /prometheus/api/v1/alerts
  hosts:
    - <hostname>
  tls:
    - secretName: host-tls
      hosts:
        - <hostname>
compactor:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""
    annotations: {}
    claims:
      - name: data
        size: 10Gi
        storageClass: ""
indexGateway:
  enabled: true
  persistence:
    enabled: true
    inMemory: false
    size: 10Gi
    storageClass: ""
    annotations: {}
memcachedChunks:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""
memcachedFrontend:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""
memcachedIndexQueries:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""
memcachedIndexWrites:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    storageClass: ""

Chunks and index are getting created as well.

above is my values.yaml file, could anyone please help me, what I have configured wrong ? I want to see all logs from the retention period is set on grafana dashboard

[JStickler]

Configuration questions have a better chance of being answered if you ask them on the community forums. More people monitor that channel.

I was searching for configuration settings with a 3 hour default, and found this in the Upgrade docs: "query_ingesters_within under the querier config now defaults to 3h, previously it was 0s. Any query (or subquery) that has an end time more than 3h ago will not be sent to the ingesters, this saves work on the ingesters for data they normally don't contain. If you regularly write old data to Loki you may need to return this value to 0s to always query ingesters." https://grafana.com/docs/loki/latest/setup/upgrade/#loki-6