search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
22.78k stars 3.31k forks source link

invalid tsdb path #10855

Open nektarios-d opened 9 months ago

nektarios-d commented 9 months ago

Describe the bug After switching to TSDB on 11/10/23 started getting the following error 20 minutes later level=error ts=2023-10-11T12:09:02.495281457Z caller=table_manager.go:133 index-store=tsdb-2023-10-11 msg="error syncing local boltdb files with storage" err="failed to sync index set for table loki_index_19641: invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602"

Restarting the container resulted in the container restarting multiple times and failing to start.

2023-10-11 15:09:25.365 | level=warn ts=2023-10-11T12:09:25.268636824Z caller=loki.go:288 msg="global timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default global timeout (\"5m\")." |  
  |   | 2023-10-11 15:09:25.365 | level=warn ts=2023-10-11T12:09:25.276421625Z caller=cache.go:127 msg="fifocache config is deprecated. use embedded-cache instead" |  
  |   | 2023-10-11 15:09:25.365 | level=warn ts=2023-10-11T12:09:25.276566077Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - frontend.embedded-cache" |  
  |   | 2023-10-11 15:09:25.365 | level=warn ts=2023-10-11T12:09:25.277828775Z caller=cache.go:127 msg="fifocache config is deprecated. use embedded-cache instead" |  
  |   | 2023-10-11 15:09:25.365 | level=warn ts=2023-10-11T12:09:25.277850555Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.511790415Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512343743Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696983300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696983300" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512453685Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512669768Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986000" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512753799Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986900" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.51281315Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696987800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696987800" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512884171Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696989600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696989600" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512934892Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696990500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696990500" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.512982952Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696991400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696991400" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513034043Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696993200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696993200" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513091104Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696994100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696994100" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513139715Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995000" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513182675Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995900" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513228606Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696997700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696997700" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513315797Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696998600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696998600" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513397368Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696999500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696999500" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513448789Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697000400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697000400" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.51349439Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697001300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697001300" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.51353916Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513588471Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697003100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697003100" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513644562Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697004900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697004900" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513700143Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697005800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697005800" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513743203Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697007600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697007600" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513789224Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697008500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697008500" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513830994Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697009400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697009400" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513892695Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697010300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697010300" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.513942346Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697011200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697011200" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514043257Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697012100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697012100" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514093598Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697013000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697013000" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514139079Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697014800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697014800" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.51419494Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697015700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697015700" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.51424488Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697016600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697016600" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514290101Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697017500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697017500" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514335582Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697019300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697019300" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514381292Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697020200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697020200" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514424253Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697021100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697021100" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514471323Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022000" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514518894Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022900" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514566115Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800" |  
  |   | 2023-10-11 15:09:25.564 | level=error ts=2023-10-11T12:09:25.514610595Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697024700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697024700" |  
  |   | 2023-10-11 15:09:25.662 | invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800 |  
  |   | 2023-10-11 15:09:25.662 | error initialising module: store |  
  |   | 2023-10-11 15:09:25.662 | github.com/grafana/dskit/modules.(*Manager).initModule |  
  |   | 2023-10-11 15:09:25.662 | /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138 |  
  |   | 2023-10-11 15:09:25.662 | github.com/grafana/dskit/modules.(*Manager).InitModuleServices |  
  |   | 2023-10-11 15:09:25.662 | /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 |  
  |   | 2023-10-11 15:09:25.662 | github.com/grafana/loki/pkg/loki.(*Loki).Run |  
  |   | 2023-10-11 15:09:25.662 | /src/loki/pkg/loki/loki.go:461 |  
  |   | 2023-10-11 15:09:25.662 | main.main |  
  |   | 2023-10-11 15:09:25.662 | /src/loki/cmd/loki/main.go:110 |  
  |   | 2023-10-11 15:09:25.662 | runtime.main |  
  |   | 2023-10-11 15:09:25.662 | /usr/local/go/src/runtime/proc.go:250 |  
  |   | 2023-10-11 15:09:25.662 | runtime.goexit |  
  |   | 2023-10-11 15:09:25.662 | /usr/local/go/src/runtime/asm_amd64.s:1598 |  
  |   | 2023-10-11 15:09:25.662 | level=error ts=2023-10-11T12:09:25.612792331Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800" |  
  |   | 2023-10-11 15:09:25.662 | level=error ts=2023-10-11T12:09:25.658745999Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800" |  
  |   | 2023-10-11 15:09:25.662 | level=error ts=2023-10-11T12:09:25.65880354Z caller=index_set.go:103 table-name=loki_index_19641 msg="failed to initialize table loki_index_19641, cleaning it up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800" |  
  |   | 2023-10-11 15:09:25.662 | level=error ts=2023-10-11T12:09:25.662459102Z caller=log.go:230 msg="error running loki" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598" |  
  |   | 2023-10-11 15:09:27.035 | level=warn ts=2023-10-11T12:09:26.935923666Z caller=loki.go:288 msg="global timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default global timeout (\"5m\")." |  
  |   | 2023-10-11 15:09:27.035 | level=warn ts=2023-10-11T12:09:26.943764238Z caller=cache.go:127 msg="fifocache config is deprecated. use embedded-cache instead" |  
  |   | 2023-10-11 15:09:27.035 | level=warn ts=2023-10-11T12:09:26.943807699Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - frontend.embedded-cache" |  
  |   | 2023-10-11 15:09:27.035 | level=warn ts=2023-10-11T12:09:26.946578679Z caller=cache.go:127 msg="fifocache config is deprecated. use embedded-cache instead" |  
  |   | 2023-10-11 15:09:27.035 | level=warn ts=2023-10-11T12:09:26.946616959Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.177696198Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.177813909Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696983300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696983300" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.17786913Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.177928861Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986000" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.178014552Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696986900" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.178066213Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696987800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696987800" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.178118554Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696989600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696989600" |  
  |   | 2023-10-11 15:09:27.235 | level=error ts=2023-10-11T12:09:27.178164304Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696990500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696990500" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178211755Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696991400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696991400" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178266636Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696993200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696993200" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178322917Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696994100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696994100" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178369007Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995000" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178413958Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696995900" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178455698Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696997700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696997700" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178499279Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696998600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696998600" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.1785439Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696999500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696999500" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.17858799Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697000400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697000400" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178631221Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697001300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697001300" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178683232Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178726942Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697003100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697003100" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178780783Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697004900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697004900" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178830514Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697005800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697005800" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.178888325Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697007600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697007600" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.183841445Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697008500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697008500" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184014378Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697009400, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697009400" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184075829Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697010300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697010300" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184133289Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697011200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697011200" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.18418319Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697012100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697012100" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184239761Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697013000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697013000" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184294102Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697014800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697014800" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184347813Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697015700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697015700" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184392883Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697016600, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697016600" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184438144Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697017500, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697017500" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184483054Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697019300, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697019300" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184533685Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697020200, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697020200" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184575856Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697021100, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697021100" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184617406Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022000, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022000" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184659147Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022900, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697022900" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184703278Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697023800" |  
  |   | 2023-10-11 15:09:27.236 | level=error ts=2023-10-11T12:09:27.184756718Z caller=index_set.go:132 table-name=loki_index_19641 msg="failed to open existing index file /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697024700, removing the file and continuing without it to let the sync operation catch up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697024700" |  
  |   | 2023-10-11 15:09:27.435 | level=error ts=2023-10-11T12:09:27.402551267Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602" |  
  |   | 2023-10-11 15:09:27.477 | invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200 |  
  |   | 2023-10-11 15:09:27.477 | error initialising module: store |  
  |   | 2023-10-11 15:09:27.477 | github.com/grafana/dskit/modules.(*Manager).initModule |  
  |   | 2023-10-11 15:09:27.477 | /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138 |  
  |   | 2023-10-11 15:09:27.477 | github.com/grafana/dskit/modules.(*Manager).InitModuleServices |  
  |   | 2023-10-11 15:09:27.477 | /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108 |  
  |   | 2023-10-11 15:09:27.477 | github.com/grafana/loki/pkg/loki.(*Loki).Run |  
  |   | 2023-10-11 15:09:27.477 | /src/loki/pkg/loki/loki.go:461 |  
  |   | 2023-10-11 15:09:27.477 | main.main |  
  |   | 2023-10-11 15:09:27.477 | /src/loki/cmd/loki/main.go:110 |  
  |   | 2023-10-11 15:09:27.477 | runtime.main |  
  |   | 2023-10-11 15:09:27.477 | /usr/local/go/src/runtime/proc.go:250 |  
  |   | 2023-10-11 15:09:27.477 | runtime.goexit |  
  |   | 2023-10-11 15:09:27.478 | /usr/local/go/src/runtime/asm_amd64.s:1598 |  
  |   | 2023-10-11 15:09:27.478 | level=error ts=2023-10-11T12:09:27.475812036Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200" |  
  |   | 2023-10-11 15:09:27.478 | level=error ts=2023-10-11T12:09:27.475874016Z caller=index_set.go:103 table-name=loki_index_19641 msg="failed to initialize table loki_index_19641, cleaning it up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200" |  
  |   | 2023-10-11 15:09:27.478 | level=error ts=2023-10-11T12:09:27.477630771Z caller=log.go:230 msg="error running loki" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696984200\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

After some tries it managed to start but the logs where flooded with

level=error ts=2023-10-11T12:35:43.236921257Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1696982602"
level=error ts=2023-10-11T12:35:43.352668638Z caller=index_set.go:285 table-name=loki_index_19641 msg="sync failed, retrying it" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200"
level=error ts=2023-10-11T12:35:43.352706559Z caller=index_set.go:103 table-name=loki_index_19641 msg="failed to initialize table loki_index_19641, cleaning it up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200"
level=error ts=2023-10-11T12:35:43.354659562Z caller=table.go:324 table-name=loki_index_19641 msg="failed to init user index set " err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200"
level=error ts=2023-10-11T12:35:43.354703093Z caller=table.go:342 table-name=loki_index_19641 org_id=fake traceID=71832575adf72359 msg="index set  has some problem, cleaning it up" err="invalid tsdb path: /data/loki/tsdb-cache/loki_index_19641/65867a84fe18-1675329762671420908-1697002200"

Changed storage config from

storage_config:
  tsdb_shipper:
    active_index_directory: /data/loki/tsdb-index
    cache_location: /data/loki/tsdb-cache

To

storage_config:
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache

Restarted the container again and the errors went away.

To Reproduce Steps to reproduce the behavior: 1) Changed from boltdb to tsdb. 2) Restarted Loki after getting errors 3) Loki fails to start and then gets flooded with errors

Expected behavior Loki to switch to TSDB without errors.

Environment:

Config Changes

storage_config:
  tsdb_shipper:
    active_index_directory: /data/loki/tsdb-index
    cache_location: /data/loki/tsdb-cache
    cache_ttl: 24h                           # default
    shared_store: s3

schema_config:
  configs:
    - from: 2023-01-01
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
    - from: "2023-10-11"
      index:
        period: 24h
        prefix: loki_index_
      object_store: aws
      schema: v12 # Current recommended schema version
      store: tsdb
JStickler commented 8 months ago

@nektarios-d I'm unclear as to why you opened this issue, this looks like a configuration error that you managed to solve yourself.

DylanGuedes commented 8 months ago

did you added the "2023-10-11" schema config before "2023-10-11"? otherwise Loki would write some data in boltdb format and only then would move to tsdb, which would trigger the errors you saw