search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.85k stars 3.44k forks source link

Loki error no space left on device #11538

Open xzsamit opened 10 months ago

xzsamit commented 10 months ago

Loki error no space left on device, but my Utilization disk /data high avail.

4|loki     | level=error ts=2023-12-20T18:03:23.174574581Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="open /data/loki/tmp/chunks/ZmFrZS8yYmIzOTY1MzM0ZjliNTk4OjE4Yzg2OGE3NzZmOjE4Yzg2OGE3NzcwOmJmNmQ5MzU4: no space left on device

image

Utilization disk /data image

tonyswu commented 10 months ago

Check your inode?

xzsamit commented 10 months ago

Check your inode?

Commands used, it can be seen that there are still quite a lot of free inodes.

df -i /data

image

tonyswu commented 10 months ago

What happens if you manually try to touch a file on the same path?

I've never tried to run Loki on a filesystem before, but this just doesn't seem like a problem with Loki.

michaelkebe commented 4 months ago

I think this is related to https://github.com/grafana/loki/issues/1502