Open RobbanHoglund opened 3 weeks ago
Also seeing this in a cluster running 3.2.0, however its only happening on one of our Read pods (Running SSD via Helm), the same pod is reporting High query latency (upto 50s), and reporting a fair amount of context cancelled in the logs, All the other read pods are behaving just fine and returning query in sub-seconds
I ran into something similar with 3.1.1 . Restarting the broken pod seem to have fixed that.
Indeed, I ran into this issue is 3.2.0
and killing the pod and spin up a new one fixed it
I'm having the same issue with Loki 3.2.0 deployed using Docker and SSD. Once in a while Read container return msg="failed mapping AST" err="context canceled" errors, only way to resolve is to restart the read container. I don't spot any issues in metrics...
Describe the bug After upgrade to Loki 3.2.0 it is repeatedly logging this for some queries: ts=2024-10-04T01:09:28.649297614Z caller=spanlogger.go:111 middleware=QueryShard.astMapperware org_id=fake traceID=35d59f6204919421 user=fake caller=log.go:168 level=warn msg="failed mapping AST" err="context canceled" query="{application=\"myapp\",level=~\"INFO|WARN|ERROR\"} "
To Reproduce Steps to reproduce the behavior:
Expected behavior If this is an actual problem in Loki 3.2.0 it would be good to get more information about what is the root cause.
Environment:
Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem.