rpc error: code = Unknown desc = failed to get s3 object: NoSuchKey: The specified key does not exist. status code: 404

[teloon999]

Using MinIO as storage, I lost some chunk and index files, resulting in an error querying historical data using grafana: "rpc error: code = Unknown desc = failed to get s3 object: NoSuchKey: The specified key does not exist. status code: 404, request id: 1711F0371EFA9FC2, host id:",

Below is my configuration information：

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  max_chunk_age: 24h
  chunk_target_size: 1572864
  chunk_idle_period: 5h
  chunk_retain_period: 30s

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: s3

storage_config:
  aws:
    bucketnames: loki
    endpoint: http://192.168.xx.xx:9000
    s3forcepathstyle: true
    access_key_id: xxxx
    secret_access_key: xxxx
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: s3

schema_config:
  configs:
    - from: 2022-08-28
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

please help me, how to query the previous data normally, thank you very much!

[GNSunny]

I have the same error

org_id=fake msg="error fetching chunks" err="failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id:

Environment:

Infrastructure: Kubernetes 1.23
Deployment tool: Helm
Loki version: 2.7.3

Configuraiton:

loki:
    storage:
      bucketNames:
        chunks: dev-loki
        ruler: dev-loki
        admin: dev-loki
      type: s3
      s3:
        region: eu-west-1
    schemaConfig:
      configs:
        - from: 2020-11-05
          store: boltdb-shipper
          object_store: s3
          schema: v11
          index:
            prefix: loki_index_
            period: 24h
        - from: 2023-06-01
          store: boltdb-shipper
          object_store: s3
          schema: v12
          index:
            prefix: loki_index_
            period: 24h
singleBinary:
    replicas: 1

I have seen many logs with

org_id=fake msg="error fetching chunks" err="failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id:

any help would be much appreciated.

[erickeller]

also have some minio S3 setup which unfortunately had some unexpected downtime leading to this exact error name:

version:

loki, version 2.7.4 (branch: HEAD, revision: 98421b0c0)
  build user:       root@6e65614e28c8
  build date:       2023-02-24T17:11:28Z
  go version:       go1.20.1
  platform:         linux/amd64

error:

Mar 16 18:52:43 maas loki[916755]: level=error ts=2023-03-16T17:52:43.733849692Z caller=batch.go:720 org_id=fake msg="error fetching chunks" err="failed to get s3 object: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 174CF7BF3E0BD838, host id: "

configuration:

auth_enabled: false

server:
  http_listen_port: 31000
  grpc_listen_port: 9096
  http_server_read_timeout: 300s
  http_server_write_timeout: 300s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h

common:
  path_prefix: /var/lib/loki
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

storage_config:
  boltdb_shipper:
    active_index_directory: /var/lib/loki/index
    cache_location: /var/lib/loki/index_cache
    shared_store: s3
  aws:
    s3: http://foo:bar@minio.example.com:8080/loki
    s3forcepathstyle: true
    region: eu-west-1
    insecure: true

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_concurrent_tail_requests: 20
  max_cache_freshness_per_query: 10m

compactor:
  working_directory: /var/lib/loki/data/compactor
  shared_store: s3
  compaction_interval: 5m

ruler:
  alertmanager_url: http://localhost:9093

Any idea how to debug this/ fix this by removing the error chunks heal loki?

[CaesarDong]

I have the same error, after using minio client "mc rm" to remove objects.

error msg:

 failed to get s3 object: NoSuchKey: The specified key does not exist.
 status code: 404, request id: 175746DA1FDD0D27, host id: 

How to sync loki & minio storage objects？