docmerlin
commented
3 years ago If you don't care about data outside the window then your alert should stop keeping data so old, right now it keeps quite a bit of data in its history to prevent flapping etc.
You don't need a barrier, unless new data for a tagset stops coming in.
Are you using docker?
The carnality does fluctuate and stay within a reasonable size when a tick script is example, as the various aforementioned outputs show. So why the growth till OOM of Kapacitor?
- I don't know for your example.
Since count is always 1, you should use a |count() instead of your |sum("count")
Your cardinality isn't insane, so you shouldn't be using anywhere near that much memory. We did slightly reduce memory pressure in 1.5.8, an upgrade may mitigate your problem signicantly. Also, if that doesn't work try adding the environmental variable GODEBUG=madvdontneed=1
Good luck, if this doesn't work, come back to me and we can try deeper debugging
The change in 1.5.8 significantly reduced memory pressure from github.com/influxdata/kapacitor/edge.(*pointMessage).GroupInfo
Good luck, if this doesn't work, feel free to poke me again.
joshdurbin
conet
Overview
I'm trying to leverage Kapacitor 1.5.7 on Linux/amd64 for context aware traffic alerting in our multi-tenant commerce system at our ingress points for traffic spikes, etc... Typically this means data collection on two (optionally three -- in this example the data is sent but not grouped) fields:
We collect and report on the data in 1-2 minute windows and don't care about any data outside such window. That is, if the point in question is over 2 minutes old, it should be expired and expunged. Kapacitor is a standalone Influx product for this use case -- there is no InfluxDB instance for this data, no retention policies, etc... Data is transmitted to Kapacitor via the UDP listener.
The format of the message is the following:
As shown in singled-out stream stats later detailed in this issue, the cardinality of each of the aforementioned fields is roughly:
uri- unknown, mediumid- 20-30k within a minute window, avgip- 40-60k within a minute window, avgrole- at most 3The
countparameter exists so that we can run a mathematical sum operation on the data in the pipe. However, this is redundant because each request generates its own message, its own point, always.We found Kapacitor struggling with unbounded memory growth in our Production systems, something we did not observe in other (non live traffic) environments. Our initial response to these uncontrollable runaway memory situations was to examine and reduce the cardinality of sets, particularly group by operations on streams. We initially tried reporting on the IP address, the Store ID, and the URI. These are all relatively high cardinality fields, but putting them together in an ordered group by wasn't helping with efficient and unbounded use of memory. So, we paired things back to the following tick script where the
uriis dropped from the equation:The script is straight forward enough; we group on the stream by
ip, thenid, from thecombinedmeasurement. A barrier exists to delete data after one minute. These operations are assigned to a stream variable which is used in alerting to do different things (at the same threshold).The dot graph and sample output of that tick script while running renders:
As previously mentioned, what we saw with this is that over time (pretty quickly) we ran out of memory. The following graph shows various tweaks to the aforementioned script changing things like the window and barrier periods didn't seem to make any difference to how fast the script/pipeline/Kapacitor consumed memory.
The various spikes in memory show me altering the tick script, removing the window, removing the barrier, changing the barrier from idle to period, changing the time of the barrier tick / window, etc... During these iterations I collected data. The data below is from engagement of the aforementioned tick script, with only changes to the window and barrier periods.
Heap dumps show the following for
in use objects:and for
in use space:A profile dump at roughly the same time shows:
Perplexed I decided to chop things up and create two tick scripts instead that monitor each of those metrics independently. The first,
top_ipsdoes no variable assignment in the tick script and things are piped together in a single flow. The second,top_storeshas assignment and piping such that data streams to two alerts that do slightly different things with those triggers, like the aforementioned combined script.Data to the measurement
ipslooks like:Here's the show output for
top_ips:... and for top stores the data looks like:
with a evaluated script like:
Note the cardinality of these, at least at the time sampled, was ~12K for store IDs and ~34k for IPs. These on their own seem small potatoes, and, even in the combined script where a group by splits first by the IP, then the Store, shouldn't be too much data for a one or two minute window.
At first this seemed to be a more stable approach, memory didn't seem to grow as fast and I thought we'd level off. Unfortunately, as the graph shows below, we did not.
Heap dumps show the following for
in use objects:and for
in use space:A profile dump at roughly the same time shows:
So I'm left head scratching of running through many iterations of changes to the tick scripts. The data, while constantly flowing the Kapacitor doesn't cause memory budge from about a gigabyte if all the tick scripts are disabled / inhibited from processing.
A few questions:
sumdoesn't really seem to be anywhere near the problem with this configuration and the growth issue, is there a cleaner way of identifying the count of points in the stream? Is it possible for me to drop the valuecount=1given that's the only purpose it serves?I've attached (top_combined_and_top_ip_and_store_kapacitor_1.5.7_growth.tar.gz) the various dumps / profile captures, etc... from the time each of these steps were in place, an output of the structure is:
With the files in
top_ip_and_store_idandtop_ip_and_store_id_lasttaken a few hours apart as the ^^^ shows.