panic: runtime error: slice bounds out of range [-2:]

grafana / loki

Like Prometheus, but for logs.

https://grafana.com/loki

GNU Affero General Public License v3.0

23.97k stars 3.46k forks source link

panic: runtime error: slice bounds out of range [-2:] #15029

Closed darkelf21cn closed 1 day ago

darkelf21cn commented 2 days ago

Describe the bug Loki crashes when compactor start to work

To Reproduce

Start loki
Wait for the compactor to start
Crash happens

Expected behavior compactor works normally

Environment:

Infrastructure: Kubernetes
Deployment tool: helm

Screenshots, Promtail config, or terminal output config.txt loki.log

darkelf21cn commented 1 day ago

I think I found the root cause. I should not use the same directory for boltdb_shipper and tsdb_shipper. Which mess up the directory and cause the compactor to load unexpected files. Everything works fine after I removed bad configurations below.

boltdb_shipper:
  active_index_directory: /loki/index
  cache_location: /loki/index_cache
tsdb_shipper:
  active_index_directory: /loki/index
  cache_location: /loki/index_cache