logdna / logdna-agent-v2

The blazingly fast, resource efficient log collection client
https://logdna.com
MIT License
65 stars 47 forks source link

Include filename in `logdna_agent_fs_lines` and other fs module prometheus metric #499

Open nmiculinic opened 1 year ago

nmiculinic commented 1 year ago

Currently the logdna_agent_fs_lines is a simple counter without any labels. For the purposes of observing the logging agent properly, it'd make things simpler to add additional labels to the metric, similar to promtail agent:

Promtail include the filepath at least.

While I'd love the k8s enrichment to be included as well (pod, namespace, etc. in the labels of that metric), seeming like k8s things are being handled later in a middleware https://github.com/nmiculinic/logdna-agent-v2/blob/9b3763b10f5b893ad4cc634e4fc35db1ec973942/bin/src/_main.rs#L260 and isn't really present at the point of metric insertion: https://github.com/nmiculinic/logdna-agent-v2/blob/df491cd8cca74d98cc6d8ca7ce25d559073c602c/common/k8s/src/event_source.rs#L403

With some regex (e.g this is container log path):

/var/log/containers/<pod_name>_<namespace>-<random suffix>.log

it's possible in post-processing to determine k8s namespace of the log line being scrape

dkhokhlov commented 1 year ago

@nmiculinic

add additional labels to the metric, similar to promtail agent

in that link I see it adds some counters but not additional metadata.?

nmiculinic commented 1 year ago

@dkhokhlov exactly