search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.8k stars 3.43k forks source link

Old data is not being deleted. #9821

Open Dolbodrynya opened 1 year ago

Dolbodrynya commented 1 year ago

Describe the bug Old data is not being deleted.

To Reproduce

Expected behavior The expectation is that data with the label selector '{unit="NetworkManager.service"}' will be deleted after 24 hours, but this is not happening.

Environment:

Screenshots, Promtail config, or terminal output

auth_enabled: false

server:
  http_listen_port: 3100
  log_level: debug
  # Read timeout for HTTP server
  http_server_read_timeout: 3m
  # Write timeout for HTTP server
  http_server_write_timeout: 3m

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 3m
  chunk_retain_period: 30s
  max_transfer_retries: 0
  wal:
    enabled: true
    dir: /loki/wal

query_range:
  split_queries_by_interval: 0
  parallelise_shardable_queries: false

querier:
  max_concurrent: 2048

frontend:
  max_outstanding_per_tenant: 4096
  compress_responses: true

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 672h
    - from: 2023-06-19
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 4392h
    - from: 2023-06-20
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: 2023-06-22
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index2_
        period: 24h
    - from: 2023-06-23
      store: tsdb
      object_store: aws
      schema: v12
      index:
        prefix: loki_  
        period: 24h

storage_config:
  boltdb:
    directory: /data/loki/index

  boltdb_shipper:
    active_index_directory: /data/loki/bs_index
    cache_location: /data/loki/bs_index_cache

  filesystem:
    directory: /data/loki/chunks

  tsdb_shipper:
    active_index_directory: /data/loki/tsdb-index
    cache_location: /data/loki/tsdb-cache
  aws:
    bucketnames: loki
    endpoint: http://ceph.local:7480
    region: default
    access_key_id: ***
    secret_access_key: ***
    insecure: true
    sse_encryption: false
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 0s
      insecure_skip_verify: false
    s3forcepathstyle: true

compactor:
  working_directory: /data/loki/retention
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 24h
  retention_delete_worker_count: 150

limits_config:
  ingestion_rate_strategy: local # Default: global
  max_global_streams_per_user: 5000
  max_query_length: 0h # Default: 721h
  max_query_parallelism: 32 # Old Default: 14
  max_streams_per_user: 0 # Old Default: 10000
  enforce_metric_name: false
  reject_old_samples: false
  reject_old_samples_max_age: 4392h
  retention_period: 24h
  retention_stream:
  - selector: '{namespace="prod"}'
    priority: 4
    period: 4392h
  - selector: '{namespace="uat"}'
    priority: 3
    period: 168h
  - selector: '{namespace="dev"}'
    priority: 2
    period: 168h
  - selector: '{container="loki"}'
    priority: 1
    period: 168h
  - selector: '{namespace="default"}'
    priority: 5
    period: 24h
  - selector: '{unit="NetworkManager.service"}'
    priority: 6
    period: 24h

chunk_store_config:
  max_look_back_period: 0s
dev-ago commented 1 year ago

We have the same problem. There is no way to delete the logs older than 120 days that we have configured. Loki seems to ignore this error.

Loki Version: 

loki, version 2.6.1 (branch: HEAD, revision: 6bd05c9a4)
  build user: root@af90ed01061f
  build date: 2022-07-18T08:41:09Z
  go version: go1.17.6
  platform: linux/amd64

Loki Config:

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_server_max_recv_msg_size: 41943040
  register_instrumentation: false
  grpc_server_max_send_msg_size: 41943040
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  wal:
    dir: /data/loki/walschema_config:
  configs:
  - from: 2018-04-15
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 168h
  - from: 2022-09-16
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
storage_config:
  boltdb:
    directory: /data/loki/index  filesystem:
    directory: /data/loki/chunkslimits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 2880hchunk_store_config:
  max_look_back_period: 0table_manager:
  chunk_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  index_tables_provisioning:
    inactive_read_throughput: 0
    inactive_write_throughput: 0
    provisioned_read_throughput: 0
    provisioned_write_throughput: 0
  retention_deletes_enabled: true
  retention_period: 2880h
analytics:
  reporting_enabled: falseruler:
  storage:
    type: local
    local:
      directory: /data/loki/rules
  rule_path: /data/loki/rules-temp
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  wal:
    dir: /data/loki/ruler-walcompactor:
  working_directory: /data/loki/retention
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
dev-ago commented 1 year ago

We solve the problem with an update to the version 2.9 of loki and this config:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_server_max_recv_msg_size: 41943040
  register_instrumentation: false
  grpc_server_max_send_msg_size: 41943040

common:
  instance_addr: 127.0.0.1
  path_prefix: /data/loki
  storage:
    filesystem:
      chunks_directory: /data/loki/chunks
      rules_directory: /data/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

ingester:
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  wal:
    dir: /data/loki/wal

query_scheduler:
  max_outstanding_requests_per_tenant: 32768

querier:
  max_concurrent: 16

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 512

schema_config:
  configs:
  - from: 2018-04-15
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 168h
  - from: 2022-09-16
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
  - from: 2023-09-13
    store: tsdb
    object_store: filesystem
    schema: v12
    index:
      prefix: tsdb_index_
      period: 24h

storage_config:
  boltdb:
    directory: /data/loki/index

  tsdb_shipper:
    active_index_directory: /data/loki/tsdb-index
    cache_location: /data/loki/tsdb-cache
    shared_store: filesystem

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 2880h

chunk_store_config:
  max_look_back_period: 2880h

table_manager:
  retention_deletes_enabled: true
  retention_period: 2880h

analytics:
  reporting_enabled: false

ruler:
  enable_api: true
  wal:
    dir: /data/loki/ruler-wal

compactor:
  working_directory: /data/loki/retention
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  delete_request_cancel_period: 5m
  retention_table_timeout: 1h
  retention_delete_delay: 5m
  retention_delete_worker_count: 150
dev-ago commented 1 year ago

Ticket can be closed!