search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.99k stars 3.46k forks source link

error pushing logs on some of promtails #14566

Open Nello-Angelo opened 1 month ago

Nello-Angelo commented 1 month ago

i have cluster kubernetes with 5 workers 3 of 5 promtails send logs correct without errors but 2 of 5 show this errors

i have loki-distr chart version 0.78.5 and promtail chart version 6.16.6

level=warn ts=2024-10-21T23:11:33.342452717Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:11:44.32676373Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:11:55.6154072Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:12:08.096187322Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:12:22.099635332Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:12:41.300830537Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:13:13.319486633Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:14:01.574312079Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:15:24.88058367Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:17:47.188632385Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=error ts=2024-10-21T23:17:47.188688978Z caller=client.go:430 component=client host=loki.gitaas.ru msg="final error sending batch" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:17:57.319603211Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:18:08.172302983Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:18:19.90990088Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:18:32.96912893Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:18:48.305290407Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:19:08.800248166Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:19:38.215865218Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-10-21T23:20:46.627935052Z caller=client.go:419 component=client host=loki.gitaas.ru msg="error sending batch, will retry" status=-1 tenant= error="Post \"https://loki.gitaas.ru/loki/api/v1/push\": context deadline exceeded
JStickler commented 3 weeks ago

I suspect this is a configuration error. Configuration questions have a better chance of being answered if you ask them on the community forums.