search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
22.71k stars 3.3k forks source link

Loki upgrade failed #13324

Open passie opened 5 days ago

passie commented 5 days ago

After upgrading to Loki 3 I'm receiving the following error messages.

invalid tsdb path: /var/loki/tsdb-cache/loki_index_19887/loki-write-0-1700653205011145856-1718247600
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
        /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
        /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
        /src/loki/pkg/loki/loki.go:453
main.main
        /src/loki/cmd/loki/main.go:122
runtime.main
        /usr/local/go/src/runtime/proc.go:267
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

and 

level=error ts=2024-06-14T13:04:28.301350256Z caller=index_set.go:310 table-name=loki_index_19887 msg="sync failed, retrying it" err="invalid tsdb path: /var/loki/tsdb-cache/loki_index_19887/loki-write-0-1700653205011145856-1718247600"
level=error ts=2024-06-14T13:04:28.301382406Z caller=index_set.go:111 table-name=loki_index_19887 msg="failed to initialize table loki_index_19887, cleaning it up" err="invalid tsdb path: /var/loki/tsdb-cache/loki_index_19887/loki-write-0-1700653205011145856-1718247600"
level=error ts=2024-06-14T13:04:28.3159973Z caller=log.go:216 msg="error running loki" err="invalid tsdb path: /var/loki/tsdb-cache/loki_index_19887/loki-write-0-1700653205011145856-1718247600\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:453\nmain.main\n\t/src/loki/cmd/loki/main.go:122\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

After manually deleting the index loki_index_19886 file the next one will pop-up. We are storing the indexes on S3 which worked fine before the upgrade. How to troubleshoot/fix this problem? Config created yesterday for the schema_config

  schemaConfig:
    configs:
      - from: 2022-01-11
        store: boltdb-shipper
        object_store: s3
        schema: v12
        index:
          prefix: loki_index_
          period: 24h
      - from: 2024-06-14
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h