search

vectordotdev / vrl

Vector Remap Language
Mozilla Public License 2.0
134 stars 65 forks source link

add `parse_envoy()` support? #50

Open parkedwards opened 2 years ago

parkedwards commented 2 years ago

https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#default-format-string

^ envoy has a bunch of different logs, i know (and the ability to custom format them, which makes adding support on the Vector side...tricky)

but if there's appetite for it, I wrote a regex string for our Vector to parse our Envoy access logs (which are their request logs) -- they're similar to what you'd see in an apache format or nginx. i imagine many envoy users are probably just using the default string format, and if so this parsing string might come in handy - and im sure you guys can do a much better job than i did.

https://regex101.com/r/pFQf5V/1

jszwedko commented 2 years ago

Thanks @parkedwards ! Similar to the functions for parsing nginx and apache logs, I think we can definitely have one for parsing Envoy's default format.

FL3SH commented 1 year ago

@parkedwards would you mind sharing a full example? I can't make it work:

$ cat program.vrl
del(.kubernetes)
del(.file)
del(.source_type)
del(.stream)
del(.timestamp)
del(.timestamp_end)
.message = string!(.message)
parse_regex!(.message,r'\[(?P<timestamp>.+?)\] "(?P<method>.+?) (?P<path>.+?) (?P<protocol>.+?)" (?P<status>.+?) (?P<resp_flags>.+?) (?P<bytes_rec>.+?) (?P<bytes_sent>.+?) (?P<duration>.+?) (.+?) "(?P<x_forwarded_for>.+?)" "(?P<user_agent>.+?)" "(?P<req_id>.+?)" "(?P<authority>.+?)" "(?P<upstream_host>.+?)"')

with input: "[2022-12-05T09:44:33.352Z] \"GET / HTTP/1.1\" 200 - via_upstream - \"-\" 0 2 0 0 \"-\" \"ELB-HealthChecker/2.0\" \"c300d0ef-498b-9b1a-8a5b-f7a7b905ce36\" \"172.16.27.220:8080\" \"172.16.27.220:8080\" inbound|8080|| 127.0.0.6:34583 172.16.27.220:8080 172.16.253.24:32888 - default"