Open tom-dell opened 4 months ago
Hi @tom-dell! I would check your Loki logs for more details about why the request was considered a bad request. I'm not sure if the Loki response would include more details but unfortunately Vector doesn't have a way to log the HTTP response bodies at the moment (there was some work started towards this in https://github.com/vectordotdev/vector/issues/15911 but we never completed it). As a workaround you could have Vector forward through a HTTP proxy where you could inspect the requests.
Seeing a similar issue - https://github.com/vectordotdev/vector/discussions/20590
Did you find what the cause of your issue was @tom-dell ?
Having similar issue https://github.com/vectordotdev/vector/discussions/20593#discussioncomment-9637102
Any updates @tom-dell ?
I also notice that after the exact same messages you get re 400 Bad Request, Loki is receiving messages from vector for about 8-9 minutes continuously, but right after that it stops receiving messages.
Update: I used Fiddler classic to debug, and found that Loki was returning 400 Bad request with reason about timestamp too new, because of timezone differences between what is being sent and what is expected. After properly using parse_timestamp, by adding the correct timezone to the date/time in log, I am no longer getting 400 bad requests.
Can confirm @tamer-hassan 's suggestion works.
Was seeing the same issue on OpenWRT (using syslog server mode as source), after applying .timestamp = format_timestamp!(.timestamp, format: "%+", timezone: "local")
to map the timezone, syslogs get ingested into Loki.
A note for the community
Problem
Hello, I'm getting some weird 400 errors from Loki when I'm trying to push my unifi logs through using Vector
you can see a syslog come in on line 59, then Vector gets a 400 from Loki, and drops the request, I cannot find any additional logs with anymore details.
Configuration
Version
timberio/vector:0.38.0-debian
Debug Output
Example Data
No response
Additional Context
Vector, Loki, and Grafana are all running in docker, all on the same docker network.
References
No response