search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.72k stars 3.43k forks source link

failed mapping AST context canceled #14384

Open RobbanHoglund opened 3 weeks ago

RobbanHoglund commented 3 weeks ago

Describe the bug After upgrade to Loki 3.2.0 it is repeatedly logging this for some queries: ts=2024-10-04T01:09:28.649297614Z caller=spanlogger.go:111 middleware=QueryShard.astMapperware org_id=fake traceID=35d59f6204919421 user=fake caller=log.go:168 level=warn msg="failed mapping AST" err="context canceled" query="{application=\"myapp\",level=~\"INFO|WARN|ERROR\"} "

To Reproduce Steps to reproduce the behavior:

  1. Started Loki 3.2.0
  2. From Grafana explore do {application="myapp",level=~"INFO|WARN|ERROR|TRACE"}

Expected behavior If this is an actual problem in Loki 3.2.0 it would be good to get more information about what is the root cause.

Environment:

Screenshots, Promtail config, or terminal output If applicable, add any output to help explain your problem. Image Image

jammiemil commented 2 weeks ago

Also seeing this in a cluster running 3.2.0, however its only happening on one of our Read pods (Running SSD via Helm), the same pod is reporting High query latency (upto 50s), and reporting a fair amount of context cancelled in the logs, All the other read pods are behaving just fine and returning query in sub-seconds

eplightning commented 4 days ago

I ran into something similar with 3.1.1 . Restarting the broken pod seem to have fixed that.

yalattas commented 4 days ago

Indeed, I ran into this issue is 3.2.0 and killing the pod and spin up a new one fixed it

vincentnonim commented 3 days ago

I'm having the same issue with Loki 3.2.0 deployed using Docker and SSD. Once in a while Read container return msg="failed mapping AST" err="context canceled" errors, only way to resolve is to restart the read container. I don't spot any issues in metrics...