search

coroot / coroot-node-agent

A Prometheus exporter based on eBPF that gathers comprehensive container metrics
https://coroot.com/docs/metrics/node-agent
Apache License 2.0
312 stars 55 forks source link

Too many errors at lunch of node #17

Closed gkalwig closed 1 year ago

gkalwig commented 1 year ago

Hi,

We have been experiencing a situation where, upon starting the k8s node, the node-agent begins producing tens of thousands of error logs for several seconds. This sudden surge in log generation is causing disruptions in our logging system's functionality.

I have look for the available configuration options, including loglvl management for the node agent, but I have not found any configuration. It seems that there is no existing configuration option to mitigate the excessive logging behavior of the node agent and no way to change loglvl of node-agent.

This issue has been negatively impacting our system's performance and hampering our ability to effectively monitor and analyze logs. We would greatly appreciate any assistance or guidance you can provide to help us resolve this problem.

image

def commented 1 year ago

@gkalwig thank you for the report. We will add a rate limiter for logging to prevent situations like that.

def commented 1 year ago

@gkalwig, both problems (too aggressive logging and support for containers within the runtime.slice cgroup) have been fixed in v1.8.3

gkalwig commented 1 year ago

@def thanks! I'll test it and get back to you next week if it's resolved.