search

vectordotdev / vrl

Vector Remap Language
Mozilla Public License 2.0
134 stars 65 forks source link

parse_vector VRL function? #946

Open davidpellcb opened 4 months ago

davidpellcb commented 4 months ago

I'm looking for a way to parse the logs emitted by Vector itself. AFAICT there isn't a parse_* function for this.

Example log:

2024-07-11T17:56:52.024579Z  WARN vector::kubernetes::reflector: Watcher Stream received an error. Retrying. error=InitialListFailed(Api(ErrorResponse { status: "Failure", message: "pods is forbidden: User \"system:serviceaccount:local-utility:vector-agent\" cannot list resource \"pods\" in API group \"\" at the cluster scope: RBAC: clusterrole.rbac.authorization.k8s.io \"vector-agent\" not found", reason: "Forbidden", code: 403 })

Is there a recommended approach for parsing Vector's logs? Just parse_grok or parse_regex? Seems like this may be hard to do. Compare this other WARN log:

2024-07-16T02:01:33.917564Z  WARN sink{component_kind="sink" component_id=proxy component_type=http}:request{request_id=1532040}: vector::sinks::util::retries: Retrying after response. reason=Http Status: 500 Internal Server Error internal_log_rate_limit=true
jszwedko commented 4 months ago

That's true, we could add this. Typically we recommend users configure Vector to log as JSON if they want to process the logs though, by setting the environment variable VECTOR_LOG_FORMAT=json.

davidpellcb commented 4 months ago

VECTOR_LOG_FORMAT=json

TIL, doh, sorry for missing that. Thanks so much! I will plan to use that and we could probably close this unless you think there's any use in being able to parse the non-JSON logs.

jszwedko commented 4 months ago

I think it's a reasonable enough request, though probably low-priority given the ability to emit structured JSON events.