How to make loki deletes logs when using SimpleScalable deployment

[yalattas]

Compator is not allowed to be used within SimpleScalable deployment model. We have set retention_streams and global retention. Yet, with streams, it lived for indefinite and never got deleted

Retention docs

https://grafana.com/docs/loki/latest/operations/storage/retention/

Our configs

  deploymentMode: SimpleScalable
  loki:
    auth_enabled: false
    compactor:
      # working_directory: /logs/retention
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: s3
    limits_config:
      retention_period: 72h
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 30m
      split_queries_by_interval: 15m
      query_timeout: 300s
      ingestion_rate_mb: 24
      ingestion_burst_size_mb: 24
      per_stream_rate_limit: "6MB"
      per_stream_rate_limit_burst: "15MB"
      volume_enabled: true
      retention_stream:
        - selector: '{retention="high"}'
          priority: 1
          period: 48h
        - selector: '{retention="low"}'
          priority: 1
          period: 24h
...
  compactor:
    replicas: 0

We're relying on promtail to add custom label to log stream and its working fine. Then in Loki, we would like to retain logs based on this custom label. Dropping is working fine as promtail won't push logs to loki. Yet, for streams that should live for 24h, 48h or more. Loki never deleted it from S3 Bucket

Promtail

promtail:
  config:
    enabled: true
    logLevel: info
    logFormat: logfmt
    serverPort: 3101
    clients:
      - url: http://loki-gateway:80/loki/api/v1/push
    snippets:
      pipelineStages:
        - cri: {}
        - match:
            selector: '{app="APP1"} |~ ".*HealthChecker/2.0.*"'
            action: drop
      extraRelabelConfigs:
        - source_labels: [__meta_kubernetes_pod_label_retention]
          target_label: retention
        - source_labels: [retention]
          target_label: retention
          action: replace
          regex: ^$
          replacement: normal
        - source_labels: [retention]
          regex: drop
          action: drop

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.

[deb0ro]

Hi, we have a same issue. Running Loki 3.1.0 installed using Helm in k8s cluster and using onprem S3 storage. The bucket just grow and grow and no retention is taking place.

Here is a part of our config:

  loki:
    auth_enabled: false
    schemaConfig:
      configs:
        - from: 2024-04-17
          store: tsdb
          object_store: s3
          schema: v13
          index:
            prefix: loki_index_
            period: 24h
    compactor:
      working_directory: /var/loki/compactor
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 2h
      retention_delete_worker_count: 150
      delete_request_store: s3
    ingester:
      chunk_encoding: snappy
      chunk_target_size: 1536000
    limits_config:
      max_global_streams_per_user: 0
      max_query_lookback: 30d
      retention_period: 30d
    storage:
      type: s3
      bucketNames:
        chunks: chunks
        ruler: ruler
        admin: admin
      s3:
        endpoint: XXX
        accessKeyId: XXX
        secretAccessKey: XXX
        insecure: false
        s3ForcePathStyle: true
        http_config:
          insecure_skip_verify: true
  deploymentMode: SimpleScalable

[duj4]

Looking forward to any further reply on this issue.

[lukasgabriel]

Same question here! It is not addressed in the docs.

[zorek187]

+1

[barnabegailliard]

+1

[xaviergrandmougin]

+1