Ratelimiting on arbitrary keys

[sinkingpoint]

Description of the problem

As a syslog-ng operator, I run many disparate services on a number of machines, all feeding their logs into syslog-ng via various sources (unix socket, journald). What this can lead to is a noisy neighbor problem where one service produces many more logs than others, which overloads the pipeline and leads to buffering and dropping of affected services.

The existing throttle option is too coarse to be useful - it affects the entire log stream, and only causes buffering rather than allowing control of logs that breach a rate limit

Proposed solution

What I would like to see is something like the Logstash Throttle Filter which allows, among other things:

• Arbitrary keys to generate a number of different rate limits
• A tagging system which allows further filters to take actions on log messages that breach the rate limit (such as dropping, or further tagging of the log lines)

Something like:

filter throttled { throttle(value("_SYSTEMD_UNIT"), 500) }
source test_source {
    junction {
        channel {
            filter(throttled);
            // Handle rate limited logs
        }
    }
}

which would have a counter for each different _SYSTEMD_UNIT and perform some action on logs that come in with a _SYSTEMD_UNIT that occurs more that 500 times a second

Additional context

I'm aware that patterndb gets us sort of? there. I guess what I'm asking is pretty similar to https://lists.balabit.hu/pipermail/syslog-ng/2011-November/017850.html (from 10 years ago) but there doesn't seem to be a conclusion there - in patterndb the correlation key (what is used to key the ratelimit) is the tuple of (pid, program, host) which might work for some usecases, but really arbitrary keys would be more useful (e.g. given a forked program with workers with different pids)

[sinkingpoint]

As for how this works for concurrency (and not destroying processing time by locking things), maybe something like having a leaky bucket per thread, with the ratelimit evenly shared between them would work

[Kokan]

Thanks again for contributing, closing the ticket as it is already merged.